Predictive virtual training systems, apparatuses, interfaces, and methods for implementing same

ABSTRACT

Apparatuses, systems, interfaces, and implementing methods including constructing training programs or routines and predictive training programs and routines implemented in a VR, AR, MR or XR environments, preparing non-predictive and/or predictive tools for use in predictive and/or non-predictive training programs or routines implemented in the VR/AR/MR/XR environments, converting non-computer assisted training programs into predictive and/or non-predictive training programs implemented in a VR and/or AR/MR/XR environments, and implementing avatars to assist trainees in performing training programs routines or any aspects thereof.

RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/US20/37807, filed 15 Jun. 2020, which claims the benefit of andpriority to U.S. Provisional Patent Application Ser. No. 62/861,971filed Jun. 14, 2019, which are incorporated by reference by operation ofthe closing paragraph.

(a) U.S. patent application Ser. No. 15/615,342 filed Jun. 6, 2017; (b)U.S. patent application Ser. No. 16/404,425 filed May 6, 2019; (c) U.S.patent application Ser. No. 16/702,266 filed Dec. 3, 2019; (d) U.S.patent application Ser. No. 15/968,872 filed May 2, 2018; (e) U.S.patent application Ser. No. 15/781,050 filed Jun. 1, 2018; (f) U.S.patent application Ser. No. 15/210,832 filed Jul. 14, 2016; (b) (g) U.S.patent application Ser. No. 15/781,057 filed Jun. 1, 2018; (h) U.S.patent application Ser. No. 15/781,061 filed Jun. 1, 2018; (i) U.S.patent application Ser. No. 16/625,553 filed Dec. 20, 2019; (b) (j) U.S.patent application Ser. No. 16/245,206 filed Jan. 10, 2019; (k) U.S.patent application Ser. No. 16/296,114 filed Mar. 7, 2019; and (1) U.S.patent application Ser. No. 16/404,382 filed May 6, 2019, which areincorporated by reference by operation of the closing paragraph.

(a) U.S. Pat. No. 10,628,977 issued Apr. 21, 2020; (b) U.S. Pat. No.10,503,359 issued Dec. 10, 2019; (c) U.S. Pat. No. 10,289,204 issued May14, 2019; (d) U.S. Pat. No. 10,263,967 issued Apr. 16, 2019; (e) U.S.Pat. No. 9,971,492 issued May 15, 2018; (f) U.S. Pat. No. 9,746,935issued Aug. 29, 2017; (g) U.S. Pat. No. 9,703,388 issued Jul. 11, 2017;(h) U.S. Pat. No. 8,788,966 issued Jul. 22, 2014; (i) U.S. Pat. No.7,861,188 issued Dec. 28, 2010; and (j) U.S. Pat. No. 7,831,932 issuedNov. 9, 2010, which are incorporated by reference by operation of theclosing paragraph.

BACKGROUND OF THE DISCLOSURE 1. Field of the Disclosure

Embodiments of this disclosure relate to virtual training systems,apparatuses, and/or user interfaces and methods for implementing them,wherein the virtual training systems and apparatuses include at leastone processing unit, at least one motion sensing device, at least oneinput device, and at least one display device and implementing methodsand the user interface (graphic or non-graphic) are configured toreceive input from the at least one motion sensing device and/or the atleast one input device, to receive output from the at least oneprocessing unit, and to use the input and output to invoke and interactwith training routines in virtual reality (VR) environments, augmentedreality (AR) environments, mixed reality (MR) environments, and/or anyother environment the includes both real features and computer generated(CG) features and display the training routines on the at least onedisplay device.

More particularly, embodiments of this disclosure relate to virtualtraining systems, apparatuses, and/or user interfaces and methods forimplementing them, wherein the virtual training systems and apparatusesimplementing the methods or supporting the user interface (graphic ornon-graphic) receive output from and/or include at least one processingunit, at least one motion sensing device, at least one input device, andat least one display device or other devices capable of providing userfeedback of any kind, and wherein the processing unit is configured to:receive data or input from the motion sensing devices and/or the inputdevices; use the data for training in 2D, 3D, and/or nD virtual reality(VR) environments, augmented reality (AR) environments, mixed reality(MR) environments, and/or any other environment the includes both realfeatures and computer generated (CG) features; and control, activate,interact with, and/or manipulate the environments and environmentcontent. In certain embodiment, the virtual training systems,apparatuses, and/or interfaces and methods for implementing them, permitmimicking techniques for overlaying a trainee actions onto expertactions to aid in improving trainee expertise. The virtual trainingsystems, apparatuses, and/or interfaces and methods for implementingthem, use motion-based processing to enhance virtual training andenvironment interaction, wherein the environments include virtualobjects, adjustable attributes, and hot spots (e.g., interactive points,areas, volumes, conditions, environmental features, or objects), thatmay be scrolled through, selected, and activated, and/or adjusted basedon motion data (e.g., any kind of output from a motion sensor), and/ormotion data and timed holds, and/or motion data and predictive data,and/or motion data and hard select data, and/or motion data and voicedata or tactile data, wherein the motion data may be used as the primaryinput data with the other data being used to confirm selection orvice-versa. In certain embodiments, the virtual training systems,apparatuses, and/or interfaces and methods for implementing them, mayact on the data only if one or more data properties exceeds one or morethreshold criteria associated with the particular received data.

Embodiments of this disclosure relate to training and predictive virtualtraining systems, apparatuses, and/or interfaces (i.e., trainingsystems, apparatuses, and/or interfaces including virtual or computergenerated elements), and methods for implementing them, wherein thevirtual training systems and apparatuses implementing the methods orsupporting the user interface (graphic or non-graphic) receive outputfrom and/or include at least one processing unit, at least one motionsensing device, at least one input device, and at least one displaydevice, and one or more output devices such as one or more displaydevices, wherein the processing unit includes a virtual training programand is configured to (a) output the training program in response to userinput data sensed by the sensors or received from the input devices, (b)collect user interaction data while performing the virtual trainingprogram or from other associated or non-associated inputs and/orparameters, and (c) modify, alter, change, augment, update, enhance,reformat, restructure, and/or redesign the virtual training program tobetter tailor the virtual training program for each user, for each user,type and/or for all users.

2. Description of the Related Art

Selection interfaces are ubiquitous throughout computer software anduser interface software. Most of these interfaces require motion andselection operations controlled by hard selection protocols such astapping, clicking, double tapping, double clicking, keys strokes,gestures, or other so-called hard selection protocols.

In previous applications, the inventor and inventors have describedmotion-based systems and interfaces that utilize motion and changes inmotion direction to invoke command functions such as scrolling andsimultaneously selection and activation commands. See for example U.S.Pat. Nos. 7,831,932 and 7,861,188, incorporated herein by operation ofthe closing paragraph of the specification.

More recently, the inventor and inventors have described motion-basedsystems and interfaces that utilize velocity and/or acceleration as wellas motion direction to invoke command functions such as scrolling andsimultaneously selection and activation commands. See for example U.S.Provisional Patent Application Ser. No. 61/885,453 filed Oct. 1, 2013 (1Oct. 2013).

While there are many systems and interfaces for permitting users toselect and activate a target object(s) from lists and/or sublists oftarget object(s) using movement properties, where the movementproperties act to discriminate and attract or manipulate or influencethe target object(s) or attributes of target object(s). Multiple layersof objects may have attributes changes, where the attribute of one layermay be different or to a different degree than other layers, but theyare all affected and relational in some way.

Many interfaces have been constructed to interact with, control, and/ormanipulate objects and attributes associated therewith so that a user isbetter able to view, select and activate objects and/or attributes.

Recently, motion-based interfaces have been disclosed. These interfacesuse motion as the mechanism for viewing, selecting, differentiating, andactivating virtual and/or real objects and/or attributes. However, thereis still in need in the art for improved motion-based interactions andinterfaces that present dynamic environments for viewing, selecting,differentiating, and activating virtual and/or real objects and/orattributes based on object and/or attribute properties, userpreferences, user recent interface interactions, user long terminterface interactions, or mixtures and combinations thereof.

While many systems and methods have been purposed for allowing users tointeract with virtual reality environments, there is still a need in theart for improved systems and methods for training, awareness, servicing,and mimicking systems. While many systems and methods have been purposedfor allowing users to interact with virtual reality environments, thereis still a need in the art for improved systems and methods forinteracting with VR environments or AR/MR/XR environments and elementsto improve training and to modify, alter, change, augment, update,enhance, reformat, restructure, and/or redesign a virtual trainingroutine, exercise, program, etc. based on use, collected sensor dataand/or trainee data captured during training routines, exercises,programs, etc. to better tailor the training routines, exercises,programs, etc. for each user or for all users, where the changes may beimplemented before, during and after a training session, and may eveninclude other machine learning or other parameters that can influencethe system and elements.

SUMMARY OF THE DISCLOSURE Virtual Training Systems, Apparatuses,Interfaces, and Methods Implementing Them

Embodiments of this disclosure provide methods including (a) receivingtrainee images and/or image sequences and trainer images and/or imagesequences, (b) overlaying the captured trainee images and/or imagesequence onto the capture trainer images and/or image sequences, (c)constructing a polygon fitted surface, a tessellated surface, orrendered representation of the trainee and the trainer, (d) applying thetrainee surface representation to the trainee images and/or imagesequence and the trainer surface representation to the trainer imagesand/or image sequence to construct a trainee polygon fitted, rendered,or tessellated surface images and/or polygon fitted, rendered, ortessellated surface image sequence and a trainer polygon fitted,rendered, or tessellated surface images and/or polygon fitted, rendered,or tessellated surface image sequence in the at least one processingunit, (e) time and space scaling the trainee polygon fitted, rendered,or tessellated surface images and/or polygon fitted, rendered, ortessellated surface image sequence and the trainer polygon fitted,rendered, or tessellated surface images and/or polygon fitted, rendered,or tessellated surface image sequence in the at least one processingunit, (f) overlaying the scaled trainee polygon fitted, rendered, ortessellated surface images and/or polygon fitted, rendered, ortessellated surface image sequence onto the scaled trainer polygonfitted, rendered, or tessellated surface images and/or polygon fitted,rendered, or tessellated surface image sequence in the at least oneprocessing unit, (g) comparing the trainee and the trainer scaledpolygon fitted, rendered, or tessellated surface images and/or polygonfitted, rendered, or tessellated surface image sequences to illustratedifferences, and (h) repeating the steps until the trainee movementsconform or substantially conform to the trainer movements in the atleast one processing unit. The invocation and manipulation of allaspects of the method are performed via motion-based processing withoutthe need for a hard invocation protocol such as a mouse click, a tap ona screen, or similar interrupt driven execution protocol or process.

Virtual Training Systems, Apparatuses, Interfaces, and MethodsImplementing them Including Two or More User Input Devices or SystemDevices and/or User Feedback Elements and/or Devices

Embodiments of this disclosure relate to systems, apparatuses, and/orinterfaces and methods for implementing them on, with the aid of, or ina computer, wherein the systems, apparatuses, and/or interfaces andmethods are constructed to interact with 3D or n-dimensional (nD)environments using at least two user interface elements or devices.

The systems, apparatuses, and/or interfaces include at least one motionsensor (including, in all cases herein, the output of and/or theinclusion of at least one motion senor), at least one processing unit(including, in all cases herein, the output of and/or the inclusion ofat least one processing unit), and at least two user feedback elementsand/or units (including, in all cases herein, the output of and/or theinclusion of at least two user feedback elements and/or units). Thesystems, apparatuses, and/or interfaces permit the ability to control,interact, manipulate, traverse, and/or navigate 2D, 3D, and/or nDenvironments. The feedback elements and/or units generally comprise adisplay device and an eye and/or head tracking device, wherein a cursor(herein included and throughout, a cursor is a feedback mechanism in anyform, and typically represented as a reticle or visible indicator) isincluded for controlling, interacting with, manipulating, traversing,and/or navigating a 2D, 3D, or nD environment. The feedback headtracking device may provide feedback in the form of controlling acursor, reticle or other feedback attribute on a display, device, orenvironment.

Embodiments of this disclosure provide apparatuses comprising aninterface configured to receive first input corresponding to firstmovement of at least one cursor in a handheld controller or virtualversion thereof controlling device; and receiving second inputcorresponding to moving the device within a VR or AR environmentcorresponding to the device moving in the real world; and displaying theenvironment; and/or displaying an indication that the device has movedin the real world in accord with the movement in the environment.

In other embodiments, the apparatuses further comprise the displaydevice. In other embodiments, the first input and the second input arereceived from the same input device. In other embodiments, theapparatuses further comprise the input device. In other embodiments, theinput device comprises an eye tracking device or a motion sensor. Inother embodiments, the first input is received from a first input deviceand wherein the second input is received from a second input device thatis distinct from the first input device.

Embodiments of this disclosure provide apparatuses, systems, andinterfaces and method implementing them, wherein the processing unit orprocessor collects and/or captures trainee data as the trainee isengaged in a training session and uses, modifies, alters, changes,and/or augments the training routine during or after the trainingsession. The modifications, changes, alterations, and/or augmentationsmay be performed at any time and additionally, the routine may bechanged, modified, altered, and/or augmented for each trainee.Additionally, the apparatuses, systems, and interfaces and methodimplementing them may be configured so that different versions of therouting may be saved to accommodate different needs for different groupsof trainees. For example, if the apparatuses, systems, and interfacesand method implementing them detect that a group of trainees is havingproblems mastering one particular task, then the training routineversion that these trainees would invoke would be designed to improvethe trainees competency in the task. As another example, differentversions may be constructed based on a trainees learning proclivities sothat the training routines may better accommodate the different learningpatterns of trainees emphasizing more visual features for visuallearners, more audio features for auditory learners, mixed visual andaudio features for learners that learn both visual and auditorylearners. The apparatuses, systems, and interfaces and methodimplementing them may also include tactile features to improve nonvisual, audio, or audiovisual learning.

Embodiments of this disclosure also provide predictive virtual trainingsystems, apparatuses, and/or interfaces and methods for implementingthem, wherein the systems, apparatuses, and/or interfaces receive inputfrom or include one or more processing units, one or more motion sensingdevices or motion sensors, optionally one or more non-motion sensors,one or more input devices, and one or more output devices such as one ormore display devices, wherein the processing unit includes a virtualtraining program and is configured to (a) output the training program inresponse to user input data sensed by the sensors or received from theinput devices, (b) collect user interaction data while performing thevirtual training program, and (c) modify, alter, change, augment,update, enhance, reformat, restructure, and/or redesign the virtualtraining program to better tailor the virtual training program for eachuser, for each user, type and/or for all users.

The predictive training systems, apparatuses, and/or interfaces andmethods for implementing them, wherein the virtual training systems,apparatuses, and/or interfaces include at one or more processing units,one or more motion sensing devices, one or more other sensors, one ormore input devices, and/or one or more output devices, wherein theprocessing unit is configured to: receive data from the motion sensingdevices, other sensors, and/or the input devices; use the data fortraining in 2D, 3D, 4D, and/or nD virtual reality (VR) environments orAR/MR/XR environments, and/or any other environment the includes bothreal content or features and computer generated (CG) content orfeatures; and control, activate, interact with, and/or manipulate theenvironments and environment features and/or content.

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, permit mimickingtechniques for overlaying a trainee actions onto expert actions tocompare/contrast the two and/or aid in improving trainee expertise. Thevirtual training systems, apparatuses, and/or interfaces and methods forimplementing them, use motion-based processing to enhance virtualtraining and environment interaction, wherein the environments includevirtual objects, adjustable attributes, and hot spots that may bescrolled through, selected and activated, and/or adjusted attributesbased on motion data, and/or motion data and timed holds, and/or motiondata and predictive data, and/or motion data and hard select data,and/or motion data and voice data or tactile data, wherein the motiondata may be used as the primary input data with the other data beingused to confirm selection. In certain embodiments, the virtual trainingsystems, apparatuses, and/or interfaces and methods for implementingthem, may act on the data only if one or more data properties exceedsone or more threshold criteria associated with the particular receiveddata.

The systems, apparatuses, interfaces and methods implementing them aredesigned to allow the processing unit to collect data from the sensorsand capture trainee data as the trainee trains with the training programor exercise and to use the data to modify, alter, change, augment,update, enhance, reformat, restructure, and/or redesign a virtualtraining routine, exercise, program, etc. to better tailor the trainingroutine, exercise, program, etc. for each user or for all users, wherethe changes may be implemented before, during and after a trainingsession. The processing unit is therefore configured to receive datafrom the sensors and the input devices and to use the data for trainingand/or awareness, servicing and logistics in a virtual reality (VR)environment or in an environment including real content and virtual orcomputer generated content sometime referred to as AR/MR/XR environmentsand wherein the processing unit collects data from the sensors and theinput devices and captures trainee data as the trainee interact with thetraining program or exercise and uses the data to modify, alter, change,augment, update, enhance, reformat, restructure, and/or redesign avirtual training routine, exercise, program, etc. to better tailor thetraining routine, exercise, program, etc. for each user or for allusers, where the changes may be implemented before, during and after atraining session. In certain embodiments, the virtual training andpredictive training systems, apparatuses, and/or interfaces and methodsfor implementing them of this disclosure are directed to training usersto use a device in a proper way, especially medical devices such asautomatic injector systems for administering medication for treatingdiseases, cancers, autoimmune diseases, or any other medical maladyrequiring periodic administration of medications.

BRIEF DESCRIPTION OF THE DRAWINGS OF THE DISCLOSURE

The disclosure can be better understood with reference to the followingdetailed description together with the appended illustrative drawings inwhich like elements are numbered the same:

Schematic Flow Charts Virtual Training Methods

FIG. 1A depicts an embodiment of a schematic flow chart of a method oftraining.

FIG. 1B depicts another embodiment of a schematic flow chart of a methodof training.

FIG. 1C depicts another embodiment of a schematic flow chart of a methodof training.

Virtual Training nD Systems Featuring Multiple User Input and/orFeedback Devices

FIG. 2 depicts another embodiment of an embodiment of systems,apparatuses, and/or interfaces of this disclosure using two userfeedback devices, a display device displaying a 3D environment andeye/head-tracking glasses projecting a control interface into or ontothe 3D environment such as a VR or AR environment.

FIG. 3A depicts another embodiment of an embodiment of systems,apparatuses, and/or interfaces of this disclosure using two userfeedback devices, a display device displaying a 3D environment andeye/head-tracking glasses projecting a control interface into or ontothe 3D environment such as a VR or AR environment.

FIG. 3B depicts another embodiment of an embodiment of systems,apparatuses, and/or interfaces of this disclosure using two userfeedback devices, a display device displaying a 3D environment andeye/head-tracking glasses projecting a control interface into or ontothe 3D environment such as a VR or AR environment.

FIG. 4 depicts another embodiment of an embodiment of systems,apparatuses, and/or interfaces of this disclosure using two userfeedback devices, a display device displaying a 3D environment andeye/head-tracking glasses projecting a control interface into or ontothe 3D environment such as a VR or AR environment using a 3D construct.

FIGS. 5A-D depict another embodiment of an embodiment of systems,apparatuses, and/or interfaces of this disclosure using two userfeedback devices, a display device displaying a 3D environment andeye/head-tracking glasses projecting a control interface into or ontothe 3D environment such as a VR or AR environment.

Illustrative Virtual Training System Implementations

FIGS. 6A-D depict another embodiment of an embodiment of systems,apparatuses, and/or interfaces of this disclosure using two userfeedback devices, a display device displaying a 3D environment andeye/head-tracking glasses projecting a control interface into or ontothe 3D environment such as a VR or AR environment.

FIGS. 7A-D depict another embodiment of an embodiment of systems,apparatuses, and/or interfaces of this disclosure using two userfeedback devices, a display device displaying a 3D environment andeye/head-tracking glasses projecting a control interface into or ontothe 3D environment such as a VR or AR environment.

FIGS. 8A-D depict another embodiment of an embodiment of systems,apparatuses, and/or interfaces of this disclosure using two userfeedback devices, a display device displaying a 3D environment andeye/head-tracking glasses projecting a control interface into or ontothe 3D environment such as a VR or AR environment.

FIGS. 9A-I depict another embodiment of an embodiment of systems,apparatuses, and/or interfaces of this disclosure using two userfeedback devices, a display device displaying a 3D environment andeye/head-tracking glasses projecting a control interface into or ontothe 3D environment such as a VR or AR environment.

Schematic Flow Charts Virtual Training Methods

FIG. 10 depict another embodiment of a schematic flow chart of a methodof training.

Schematic Flow Charts Predictive Virtual Training Methods

FIG. 11 depicts another embodiment of a schematic flow chart of a methodof training.

Modify Routine Based on Interaction

FIGS. 12A-D depict modifying cockpit hot spots based on userinteraction.

Interacting with the Routine Based on Motion

FIGS. 13A-J depict user interaction with the cockpit training routinevia motion-based selecting and information retrieval.

Avatar Expressions

FIGS. 14A-T depict avatar facial expressions that may be associated witha trainees performance of the routine.

FIGS. 15A-F depict avatar body expressions that may be associated with atrainees performance of the routine.

FIGS. 16A-AG depict an interaction with a system, apparatus, interface,and implementing methods being used to activate a user capture sessionillustrating the capture of the user performing a series of postures.

DEFINITIONS USED IN THE DISCLOSURE

The term “at least one”, “one or more”, and “one or a plurality” meanone thing or more than one thing with no limit on the exact number;these three terms may be used interchangeably within this application.For example, at least one device means one or more devices or one deviceand a plurality of devices.

The term “about” means that a value of a given quantity is within ±20%of the stated value. In other embodiments, the value is within ±15% ofthe stated value. In other embodiments, the value is within ±10% of thestated value. In other embodiments, the value is within ±7.5% of thestated value. In other embodiments, the value is within ±5% of thestated value. In other embodiments, the value is within ±2.5% of thestated value. In other embodiments, the value is within ±1% of thestated value.

The term “substantially” or “essentially” means that a value of a givenquantity is within ±10% of the stated value. In other embodiments, thevalue is within ±7.5% of the stated value. In other embodiments, thevalue is within ±5% of the stated value. In other embodiments, the valueis within ±2.5% of the stated value. In other embodiments, the value iswithin ±1% of the stated value. In other embodiments, the value iswithin ±0.5% of the stated value. In other embodiments, the value iswithin ±0.1% of the stated value.

The term “hard select” or “hard select protocol” or “hard selection” or“hard selection protocol” means a mouse click or double click (rightand/or left), keyboard key strike, tough down event, lift off event,touch screen tab, haptic device touch, voice command, hover event, eyegaze event, or any other action that required a user action to generatea specific output to affect a selection of an object or item displayedon a display device. The term “voice command” means an audio commandsensed by an audio sensor. The term “neural command” means a commandsensed by a sensor capable of reading neuro states.

The term “motion” and “movement” are often used interchangeably and meanmotion or movement that is capable of being detected by a motion sensorwithin an active zone of the sensor, wherein the motion may haveproperties including direction, speed, velocity, acceleration, magnitudeof acceleration, and/or changes of any of these properties over a periodof time. Thus, if the sensor is a forward viewing sensor and is capableof sensing motion within a forward extending conical active zone, thenmovement of anything within that active zone that meets certainthreshold detection criteria, will result in a motion sensor output,where the output may include at least direction, angle,distance/displacement, duration (time), velocity, and/or acceleration.Moreover, if the sensor is a touch screen or multitouch screen sensorand is capable of sensing motion on its sensing surface, then movementof anything on that active zone that meets certain threshold detectioncriteria, will result in a motion sensor output, where the output mayinclude at least direction, angle, distance/displacement, duration(time), velocity, and/or acceleration. Of course, the sensors do notneed to have threshold detection criteria, but may simply generateoutput anytime motion or any kind is detected. The processing units canthen determine whether the motion is an actionable motion or movementand a non-actionable motion or movement.

The term “motion sensor” or “motion sensing component” means any sensoror component capable of sensing motion of any kind by anything with anactive zone area or volume, regardless of whether the sensor's orcomponent's primary function is motion sensing. Of course, the same istrue of sensor arrays regardless of the types of sensors in the arraysor for any combination of sensors and sensor arrays.

The term “gaze controls” means taking gaze tracking input from sensorsand converting the output into control features including all type ofcommands. The sensors may be eye and/or head tracking sensors, where thesensor may be processors that are in communication with mobile ornon-mobile apparatuses including processors. In VR/AR/MR/XR applicationsusing mobile or non-mobile devices, the apparatuses, systems, andinterfaces of this disclosure may be controlled by input from gazetracking sensors, from processing gaze information from sensors on themobile devices or non-mobile devices or communication with the mobiledevices or non-mobile devices that are capable of determine gaze and/orposture information, or mixtures and combinations.

The term “eye tracking sensor” means any sensor capable of tracking eyemovement such as eye tracking glasses, eye tracking cameras, or anyother eye tracking sensor.

The term “head tracking sensor” means any sensor capable of trackinghead movement such as head tracking helmets, eye tracking glasses, headtracking cameras, or any other head tracking sensor.

The term “face tracking sensor” means any sensor capable of trackingface movement such as any facial head tracking gear, face trackingcameras, or any other face tracking sensor.

The term “gaze” or “pose” or “pause” means any type of fixed motion overa period of time that may be used to cause an action to occur. Thus, ineye tracking, a gaze is a fixed stare of the eyes or eye over a periodof time greater than a threshold, in body, body part, or face tracking,a pose is a stop in movement of the body or body part or holding aspecific body posture or body part configuration for a period of timegreater than a threshold, and a pause is a stop in motion for a periodof time greater than a threshold, that may be used by the systems,apparatuses, interfaces, and/or implementing methods to cause an actionto occur.

The term “real object” or “real world object” means real world device,attribute, or article that is capable of being controlled by aprocessing unit. Real objects include objects or articles that have realworld presence including physical, mechanical, electro-mechanical,magnetic, electro-magnetic, electrical, or electronic devices, waveformdevices, or any other real world device that may be controlled by aprocessing unit.

The term “virtual object” means any construct generated in or attributeassociated with a virtual world or by a computer and may be displayed bya display device and that are capable of being controlled by aprocessing unit. Virtual objects include objects that have no real worldpresence, but are still controllable by a processing unit or output froma processing unit(s). These objects include elements within a softwaresystem, product or program such as icons, list elements, menu elements,applications, files, folders, archives, generated graphic objects, 1D,2D, 3D, and/or nD graphic images or objects, generated real worldobjects such as generated people, generated animals, generated devices,generated plants, generated landscapes and landscape objects, generateseascapes and seascape objects, generated sky scapes or sky scapeobjects, 1D, 2D, 3D, and/or nD zones, 2D, 3D, and/or nD areas, 1D, 2D,3D, and/or nD groups of zones, 2D, 3D, and/or nD groups or areas,volumes, attributes or characteristics such as quantity, shape, zonal,field, affecting influence changes or the like, or any other generatedreal world or imaginary objects or attributes. Augmented and/or Mixedreality is a combination of real and virtual objects and attributes.

The term “entity” means a human or an animal or robot or robotic system(autonomous or non-autonomous or virtual representation of a real orimaginary entity.

The term “entity object” means a human or a part of a human (fingers,hands, toes, feet, arms, legs, eyes, head, body, etc.), an animal or apart of an animal (fingers, hands, toes, feet, arms, legs, eyes, head,body, etc.), or a real world object under the control of a human or ananimal or a robot and include such articles as pointers, sticks, or anyother real world object that can be directly or indirectly controlled bya human or animal or a robot. In VR/AR environments, the entity objectmay also include virtual objects.

The term “mixtures” means different objects, attributes, data, datatypes or any other feature that may be mixed together or controlledtogether.

The term “combinations” means different objects, attributes, data, datatypes or any other feature that may be packages or bundled together butremain separate.

The term “sensor data” means data derived from at least one sensorincluding user data, motion data, environment data, temporal data,contextual data, historical data, waveform data, other types of data,and/or mixtures and combinations thereof.

The term “user data” means user attributes, attributes of entities underthe control of the user, attributes of members under the control of theuser, information or contextual information associated with the user, ormixtures and combinations thereof.

The terms “user features”, “entity features”, and “member features”means features including: (a) overall user, entity, or member shape,texture, proportions, information, matter, energy, state, layer, size,surface, zone, area, any other overall feature, attribute orcharacteristic, and/or mixtures or combinations thereof; (b) specificuser, entity, or member part shape, texture, proportions,characteristics, any other part feature, and/or mixtures or combinationsthereof; (c) particular user, entity, or member dynamic shape, texture,proportions, characteristics, any other part feature, and/or mixtures orcombinations thereof; and (d) mixtures or combinations thereof. Forcertain software programs, routines, and/or elements, features mayrepresent the manner in which the program, routine, and/or elementinteract with other software programs, routines, and/or elements operateor are controlled. All such features may be controlled, manipulated,and/or adjusted by the motion-based systems, apparatuses, and/orinterfaces of this disclosure.

The term “motion data” or “movement data” means data generated by one ormore motion sensor or one or more sensors of any type capable of sensingmotion/movement comprising one or a plurality of motions/movementsdetectable by the motion sensors or sensing devices.

The term “motion properties” or “movement properties” means propertiesassociated with the motion data including motion/movement direction(linear, curvilinear, circular, elliptical, etc.), motion/movementdistance/displacement, motion/movement duration (time), motion/movementvelocity (linear, angular, etc.), motion/movement acceleration (linear,angular, etc.), motion signature or profile manner of motion/movement(motion/movement properties associated with the user, users, objects,areas, zones, or combinations of thereof), dynamic motion propertiessuch as motion in a given situation, motion learned by the system basedon user interaction with the systems, motion characteristics based onthe dynamics of the environment, influences or affectations, changes inany of these attributes, and/or mixtures or combinations thereof. Motionor movement based data is not restricted to the movement of a singlebody, body part, and/or member under the control of an entity, but mayinclude movement of one or any combination of movements of any entityand/or entity object. Additionally, the actual body, body part and/ormember's identity is also considered a movement attribute. Thus, thesystems/apparatuses, and/or interfaces of this disclosure may use theidentity of the body, body part and/or member to select betweendifferent set of objects that have been pre-defined or determined basedon environment, context, and/or temporal data.

The term “gesture” or“predetermine movement pattern” means a predefinedmovement or posture preformed in a particular manner such as closing afist lifting a finger that is captured compared to a set of predefinedmovements that are tied via a lookup table to a single function and ifand only if, the movement is one of the predefined movements does agesture based system actually go to the lookup and invoke the predefinedfunction.

The term “environment data” means data associated with the user'ssurrounding or environment such as location (GPS, etc.), type oflocation (home, office, store, highway, road, etc.), extent of thelocation, context, frequency of use or reference, attributes,characteristics, and/or mixtures or combinations thereof.

The term “temporal data” means data associated with duration ofmotion/movement, events, actions, interactions, etc., time of day, dayof month, month of year, any other temporal data, and/or mixtures orcombinations thereof.

The term “historical data” means data associated with past events andcharacteristics of the user, the objects, the environment and thecontext gathered or collected by the systems over time, or anycombinations of these.

The term “contextual data” means data associated with user activities,environment activities, environmental states, frequency of use orassociation, orientation of objects, devices or users, association withother devices and systems, temporal activities, any other content orcontextual data, and/or mixtures or combinations thereof.

The term “predictive data” means any data from any source that permitsthat apparatuses, systems, interfaces, and/or implementing methods touse data to modify, alter, change, augment, update, enhance, reformat,restructure, and/or redesign a virtual training routine, exercise,program, etc. to better tailor the training routine, exercise, program,etc. for each user or for all users, where the changes may beimplemented before, during and after a training session.

The term “simultaneous” or “simultaneously” means that an action occurseither at the same time or within a small period of time. Thus, asequence of events are considered to be simultaneous if they occurconcurrently or at the same time or occur in rapid succession over ashort period of time, where the short period of time ranges from about 1nanosecond to 5 second. In other embodiments, the period ranges fromabout 1 nanosecond to 1 second. In other embodiments, the period rangesfrom about 1 nanosecond to 0.5 seconds. In other embodiments, the periodranges from about 1 nanosecond to 0.1 seconds. In other embodiments, theperiod ranges from about 1 nanosecond to 1 millisecond. In otherembodiments, the period ranges from about 1 nanosecond to 1 microsecond.It should be recognized that any value of time between any stated rangeis also covered.

The term “and/or” means mixtures or combinations thereof so that whetheran “and/or” connectors is used, the “and/or” in the phrase or clause orsentence may end with “and mixtures or combinations thereof”.

The term “spaced apart” means for example that objects displayed in awindow of a display device are separated one from another in a mannerthat improves an ability for the systems, apparatuses, and/or interfacesto discriminate between objects based on movement sensed by motionsensors associated with the systems, apparatuses, and/or interfaces.

The term “maximally spaced apart” means that objects displayed in awindow of a display device are separated one from another in a mannerthat maximizes a separation between the objects to improve an abilityfor the systems, apparatuses, and/or interfaces to discriminate betweenobjects based on motion/movement sensed by motion sensors associatedwith the systems, apparatuses, and/or interfaces.

The term “s” means one or more seconds. The term “ms” means one or moremilliseconds (10⁻³ seconds). The terms “μs” means one or more microseconds (10⁻⁶ seconds). The term “ns” means nanosecond (10⁻⁹ seconds).The term “ps” means pico second (10⁻¹² seconds). The term “fs” meansfemto second (10⁻′^(s) seconds). The term “as” means femto second (10⁻¹⁸seconds).

The term “hold” means to remain stationary at a display location for afinite duration generally between about 1 ms to about 2 s.

The term “brief hold” means to remain stationary at a display locationfor a finite duration generally between about 1 μs to about 1 s.

The term “microhold” or “micro duration hold” means to remain stationaryat a display location for a finite duration generally between about 1 asto about 500 ms. In certain embodiments, the microhold is between about1 fs to about 500 ms. In certain embodiments, the microhold is betweenabout 1 ps to about 500 ms. In certain embodiments, the microhold isbetween about 1 ns to about 500 ms. In certain embodiments, themicrohold is between about 1 μs to about 500 ms. In certain embodiments,the microhold is between about 1 ms to about 500 ms. In certainembodiments, the microhold is between about 100 us to about 500 ms. Incertain embodiments, the microhold is between about 10 ms to about 500ms. In certain embodiments, the microhold is between about 10 ms toabout 250 ms. In certain embodiments, the microhold is between about 10ms to about 100 ms.

The term “VR” means virtual reality and encompasses computer-generatedsimulations of a two-dimension, three-dimensional and orfour-dimensional, or multi-dimensional images and/or environments thatmay be interacted with in a seemingly real or physical way by a personusing special electronic equipment, such as a helmet with a screeninside or gloves fitted with sensors.

The term “AR” means augmented reality, which is a technology thatsuperimposes a computer-generated image on a user's view of the realworld, thus providing a composite view.

The term “MR” means mixed reality is a blend of physical and virtualworlds that includes both real and computer-generated objects. The twoworlds are “mixed” together to create a realistic environment. A usercan navigate this environment and interact with both real and virtualobjects. Mixed reality (MR) combines aspects of virtual reality (VR) andaugmented reality (AR). It sometimes called “enhanced” AR since it issimilar to AR technology, but provides more physical interaction.

The term “XR” means extended reality and refers to all real-and-virtualcombined environments and human-machine interactions generated bycomputer technology and wearables. The levels of virtuality range frompartially sensory inputs to immersion virtuality, also called VR.

The terms VR is generally used to mean environments that are totallycomputer generated, while AR, MR, and XR are sometimes usedinterchangeable to mean any environment that includes real content andvirtual or computer generated content. We will often use the AR/MR/XR asa general term for all environments that including real content andvirtual or computer generated content, and these terms may be usedinterchangeably.

DETAILED DESCRIPTION OF THE DISCLOSURE

The inventors have found that systems, apparatuses, interfaces, andmethods for implementing them may be constructed including trainingroutines and environments that greatly enhance training effectivenessand efficiency by first creating new content (audio, audiovisual, video,visual, textual, image, tactile, etc.) and/or gathering existing content(audio, audiovisual, video, visual, textual, image, tactile, etc.) andstoring the content in system databases, on dedicated secure servers orsecure cloud based servers, and then generating a VR/AR/MR trainingroutine including a VR/AR/MR training environment, analyzing theenvironment and associating the content with hot spots within theenvironment, and populating the environment with the hot spots thatprovide a pathway to the hot spot content generally in the form ofactivatable objects, textual words or phrases, icons or any othercontent identifying activatable feature or computer generated (CG)construct. The content may be stored in local databases, in remotedatabases located in the “cloud” and/or on dedicated servers so that thecontent is available as a trainee traverses the training routine. Thesystems, apparatuses, interfaces, and methods for implementing theminclude tools for storing, updating, modifying, and/or changing thecontent in the databases. The content may be structured in manydifferent formats such as structuring the content according to positionor location, and/or chronologically such as on a frame by frame bases orin accord with an image sequence or based on some other timing element.The systems, apparatuses, interfaces, and methods for implementing theminclude attaching system, apparatus or interfacefunctions/attributes/analytics/logic or other software algorithmicelements with the contents so that the content andfunctions/attributes/analytics/logic or other software algorithmicelements respond to user intent and/or interactions during trainingactivities. The content may be arranged and stored hierarchically sothat once a particular content feature is activated, the systems,apparatuses, interfaces, and methods for implementing them will show thecontent in activatable object form so that the trainee may view alllevels of content using any type of technique for traversinghierarchical structure data especially using the motion-based processingof this invention, where motion alone is used to stroll and select andactive content at any level.

Once the content is positioned in space/time, and the functions andconfigurations are assigned, the systems, apparatuses, interfaces, andmethods for implementing them populates and configures the VR/AR/MRtraining environment with the content generally in hot spot objectsdistributed in the VR/AR/MR training environment according to space andtime relevancy. After constructing the VR/AR/MR training environment, atrainee perform the training allowing the trainee to interact with theconfigured VR/AR/MR training environment. The systems, apparatuses,interfaces, and methods for implementing them may be configured tooperate by touch, gaze, motion, mouse navigation, joy stick navigation,gesture navigation, and/or any combination thereof depending on thedevice being used and the format, but it could be any device running anysoftware capable of displaying a VR/AR/MR training environment andallowing a trainee to interact with the environment and all of itsfeatures and content as the trainee undergoes training using thesystems, apparatuses, interfaces, and methods for implementing them.Before, during, and/or after training, the systems, apparatuses,interfaces, and methods for implementing them analyzes the interactions,stores the training session data in the system databases, determinesperformance metrics from the session data, stores the performancemetrics in the system databases, and reports the metrics to the traineesand supervisors during and/or after training or during and/or afterperforming a given training task. The systems, apparatuses, interfaces,and methods for implementing them utilizes or are configure to use themetrics to analyze qualitative, quantitative, confidence, and/or basemetric criteria, to determine performance competency criteria, retentioncriteria, cognition criteria, and/or other performance metric criteria,and to improve, refine, and/or modify qualitative, quantitative,confidence, and/or base metric criteria, environment features, hot spotlocations, and hot spot content to improve the training experience. Thesystems, apparatuses, interfaces, and methods for implementing them mayalso use CG avatars as guides or assistants to assist the trainee inbecoming competent in each task in the training routine and to achieveoverall competency. The systems, apparatuses, interfaces, and methodsfor implementing them may fashion an avatar based on trainee input, mayuse a pre-structured CG avatar or an on the fly generated CG avatarhaving characteristic derived from the context of the training and thetask being performed. Additionally, the CG avatars may undergo changeson a task-by-task basis or a scene-by-scene basis or any combinationthereof, where the changes correlated with changes in the task or scene.The systems, apparatuses, interfaces, and methods for implementing themare configured to collected relevant training data each time thetraining routine is used and to use the collected data to improve any,some, or all aspects of one, some or all important, relevant,meaningful, and/or interesting features, attributes, properties, and/orcharacteristics associated with the training routine and all important,relevant, meaningful, and/or interesting features, attributes,properties, and/or characteristics associated with any and all equipmentused in the training routine or associated with the training routine.The systems, apparatuses, interfaces, and methods for implementing themare constructed to work seamlessly across all hardware and softwareplatforms.

Embodiments of the systems, apparatuses, interfaces and methodsimplementing them include identifying a training routine, which includesa plurality of training tasks. The methods also include developing aVR/AR/MR/XR training routine and generating a trainer avatar as anassistant for trainees participating in the routine. An Avatar may be avirtual construct of a person or thing, or may represent theview/perspective of the user or another person, or group ofperspectives. The methods also include: (a) generating information aboutall important, relevant, meaningful, and/or interesting features,attributes, properties, and/or characteristics associated with thetraining routine and all important, relevant, meaningful, and/orinteresting features, attributes, properties, and/or characteristicsassociated with any and all equipment used in the training routine orassociated with the training routine and/or (b) gathering informationabout all important, relevant, meaningful, and/or interesting features,attributes, properties, and/or characteristics associated with thetraining routine and all important, relevant, meaningful, and/orinteresting features, attributes, properties, and/or characteristicsassociated with any and all equipment used in the training routine orassociated with the training routine. The methods also include storingthe gathered information in system databases both locally and/orremotely, e.g., databases in the cloud or dedicated servers. The methodsalso include analyzing the training routine, generating a VR/AR/MRenvironment for training, and associating/populating hot spots with allrelevant, meaningful, or interesting features, attributes, properties,and/or characteristics associated with the training environment and allimportant, relevant, meaningful, and/or interesting features,attributes, properties, and/or characteristics associated with any andall equipment used in the training environment or associated with thetraining routine. The methods also include collecting/capturing expertdata as an expert goes through the training routine in the trainingenvironment, wherein the data includes data on how the expertaccomplished each training task, data on what hot spots the traineeactivates, data on the duration of time the expert spends in teach hotspot and data on the type of information viewed, data indicating a levelof competency in each task, data indicating an overall competency, anddata associated with ease of using each task, ease of using the entiretraining routine, and each of using and interacting with theenvironment. The methods also include collecting/capturing trainee dataas a trainee goes through the training routine in the trainingenvironment, wherein the data includes data on how the traineeaccomplished each training task, data on what hot spots the traineeactivates, data on the duration of time the trainee spends in teach hotspot and data on the type of information viewed, data indicating a levelof competency in each task, data indicating an overall competency, anddata associated with ease of using each task, ease of using the entiretraining routine, and each of using and interacting with theenvironment. The methods also include providing feedback to the traineeand/or to the supervisors before, during, and/or after each task orbefore, during, and/or after the entire training routine to the traineeand collecting and storing all feed back data and evaluation data madeby supervisors. The methods also include informing the trainee of anydeficiencies and providing information on how to improve the traineesperformance including showing the trainees performance overlaid on anexperts performance for the same task or suggesting way to improveperformance. The methods also include analyzing performance data andmodifying the hot spots to aid the trainee in those deficient tasks andallowing the trainee to redo the training task or the entire trainingsession. The methods also include repeating the deficient tasks or theentire routine until proficient at each task and the entire routine to agiven level of competency. The methods also include analyzing eachtraining session, determining areas that trainees are having problems,and revising the training routine, avatar, and/or environment, and basedon the analyses and determinations improving the routine, environment,avatar, hot spots, and/or hot spot content. The methods also includecontinuing the analyzing, determining, and improving the routine,environment, avatar, hot spots, and/or hot spot content, wherein theimproving may include revising, adding, deleting, modifying, and/orchanging any some or all aspects of the training routine, environment,avatar, hot spots, and/or hot spot content to enhance its effectivenessover time.

Attractive/Repulsive/Manipulative Apparatuses, Systems, and/orInterfaces

The inventor has found that selection attractive or manipulativeapparatuses, systems, and/or interfaces may be constructed that usemotion or movement within an active sensor zone of a motion sensortranslated to motion or movement of a selection object (seen or unseen)on or within a user feedback device: 1) to discriminate betweenselectable objects based on the motion, 2) to attract target selectableobjects towards the selection object based on properties of the sensedmotion including direction, angle, distance/displacement, duration,speed, acceleration, or changes thereof, and 3) to select andsimultaneously, synchronously or asynchronously activate a particular ortarget selectable object or a specific group of selectable objects orcontrollable area or an attribute or attributes upon “contact” of theselection object with the target selectable object(s), where contactmeans that: 1) the selection object actually touches or moves inside thetarget selectable object, 2) touches or moves inside an active zone(area or volume) surrounding the target selectable object, 3) theselection object and the target selectable object merge, 4) a triggeringevent occurs based on a close approach to the target selectableobject(s) or its associated active zone or 5) a triggering event basedon a predicted selection meeting a threshold certainty. The touch,merge, or triggering event causes the processing unit to select andactivate the object(s), select and activate object attribute lists,select, activate and adjustments of an adjustable attribute.

The objects may represent real and/or virtual objects including: 1) realworld devices under the control of the apparatuses, systems, orinterfaces, 2) real world device attributes and real world devicecontrollable attributes, 3) software including software products,software systems, software components, software objects, softwareattributes, active areas of sensors, 4) generated EMF fields, RF fields,microwave fields, or other generated fields, 5) electromagneticwaveforms, sonic waveforms, ultrasonic waveforms, or any other waveformor entity, and/or 6) mixture and combinations thereof. The apparatuses,systems and interfaces of this disclosure may also include remotecontrol units in wired or wireless communication therewith.

The inventor has also found that a velocity (speed and direction) ofmotion or movement or any other movement property may be used by theapparatuses, systems, or interfaces to pull or attract one or a group ofselectable objects toward a selection object and increasing speed may beused to increase a rate of the attraction of the objects, whiledecreasing motion speed may be used to slower a rate of attraction ofthe objects.

The inventors have also found that as the attracted object move towardthe selection object, they may be augmented in some way such as changedsize, changed color, changed shape, changed line thickness of the formof the object, highlighted, changed to blinking, or combinationsthereof. Simultaneously, synchronously or asynchronously, submenus orsubobjects may also move or change in relation to the movements orchanges of the selected objects. Simultaneously, synchronously orasynchronously, the non-selected objects may move away from theselection object(s). It should be noted that whenever a word object isused, it also includes the meaning of objects, and these objects may besimultaneously performing separate, simultaneous, synchronous orasynchronous, and/or combined command functions or used by theprocessing units to issue combinational functions.

Embodiments of this disclosure relate to systems, interfaces,interactive user interfaces effective for navigating large amounts ofinformation on small touchscreen devices, apparatuses including theinterfaces, and methods for implementing the systems and interfaceswhere the systems and interfaces implement a 3D control methodologyusing 2D movements, where selection attractive or manipulation systemsand interfaces use movement of in the xy plane in a ring format tosimulate 3D movement for motion-based selection and activation. The 3Dmovement methodology permits object selection and discrimination betweendisplayed objects and attract a target object, objects or groups ofobjects, or fields of objects or object attributes toward, away or atangles to or from the selection object, where the direction and speed ofmotion controls discrimination and attraction. Embodiments also includeinteractive interfaces for navigating large amounts of data,information, attributes and/or controls on small devices such aswearable smart watches, sections or areas of wearable fabric or othersensor or embedded sensor surfaces or sensing abilities, as well as inVirtual Reality (VR) or Augmented Reality (AR) environments, includingglasses, contacts, touchless and touch environments, and 2D, 3D, and/ornD (n-dimensional) environments. This more specifically, in wearabledevices, such as watches, music players, health monitors and devices,etc. allows for the control of attributes and information by sensingmotion on any surface or surfaces of the device(s), or above or aroundthe surfaces, or through remote controls. The systems may be autonomous,or work in combination with other systems or devices, such as a watch, aphone, biomedical or neurological devices, drones, etc., headphones,remote display, etc. The selection object may be a group of objects or afield, with a consistent or gradient inherent characteristic, created byany kind of waveform as well, and may be visible, an overlay ortranslucent, or partially displayed, or not visible, and may be anaverage of objects, such as the center of mass of a hand and fingers, asingle body part, multiple body and/or objects under the control of aperson, or a zone, such as an area representing the gaze of an eye(s) orany virtual representation of objects, fields or controls that do thesame.

In certain embodiments, as the selection object moves toward a targetobject, the target object will get bigger as it moves toward theselection object. It is important to conceptualize the effect we arelooking for. The effect may be analogized to the effects of gravity onobjects in space. Two objects in space are attracted to each other bygravity proportional to the product of their masses and inverselyproportional to the square of the distance between the objects. As theobjects move toward each other, the gravitational force increasespulling them toward each other faster and faster. The rate of attractionincreases as the distance decreases, and they become larger as they getcloser. Contrarily, if the objects are close and one is moved away, thegravitational force decreases, and the objects get smaller. The oppositeof these attributes may also be true, or any attributes that aredesired.

In the present disclosure, motion of the selection object away from aselectable object may act as a rest, returning the display back to theoriginal selection screen or back to the last selection screen much likea “back” or “undo” event. Thus, if the user feedback unit (e.g.,display) is one level down from the top display, then movement away fromany selectable object, would restore the display back to the main level.If the display was at some sublevel, then movement away from selectableobjects in this sublevel would move up a sublevel. Thus, motion awayfrom selectable objects acts to drill up, while motion toward selectableobjects that have sublevels results in a drill down operation. Ofcourse, if the selectable object is directly activatable, then motiontoward it selects and activates it. Thus, if the object is an executableroutine such as taking a picture, then contact with the selectionobject, contact with its active area, or triggered by a predictivethreshold certainty selection selects and simultaneously, synchronouslyor asynchronously activates the object.

Once the interface is activated, the selection object and a default menuof items may be activated on or within the user feedback unit. If thedirection of motion towards the selectable object or proximity to theactive area around the selectable object is such that the probability ofselection is increased, the default menu of items may appear or moveinto a selectable position, or take the place of the initial objectbefore the object is actually selected such that by moving into theactive area or by moving in a direction such that a commit to the objectoccurs, and simultaneously, synchronously or asynchronously causes thesubobjects or submenus to move into a position ready to be selected byjust moving in their direction to cause selection or activation or both,or by moving in their direction until reaching an active area inproximity to the objects such that selection, activation or acombination of the two occurs. The selection object and the selectableobjects (menu objects) are each assigned a mass equivalent orgravitational value of 1. The difference between what happens as theselection object moves in the display area towards a selectable objectin the present interface, as opposed to real life, is that theselectable objects only feel the gravitation effect from the selectionobject and not from the other selectable objects. Thus, in the presentdisclosure, the selectable object is an attractor, while the selectableobjects are non-interactive, or possibly even repulsive to each other.So as the selection object is moved in response to motion by a userwithin the motion sensors active zone such as motion of a finger in theactive zone the processing unit maps the motion and generatescorresponding movement or motion of the selection object towardsselectable objects in the general direction of the motion.

The processing unit then determines the projected direction of motionand based on the projected direction of motion, allows the gravitationalfield or attractive force of the selection object to be felt by thepredicted selectable object or objects that are most closely alignedwith the direction of motion. These objects may also include submenus orsubobjects that move in relation to the movement of the selectedobject(s). This effect would be much like a field moving and expandingor fields interacting with fields, where the objects inside the field(s)would spread apart and move such that unique angles from the selectionobject become present so movement towards a selectable object or groupof objects can be discerned from movement towards a different object orgroup of objects, or continued motion in the direction of the second ormore of objects in a line would cause the objects to not be selectedthat had been touched or had close proximity, but rather the selectionwould be made when the motion stops, or the last object in the directionof motion is reached, and it would be selected.

The processing unit causes the display to move those objects toward theselection object. The manner in which the selectable object moves may beto move at a constant velocity towards a selection object or toaccelerate toward the selection object with the magnitude of theacceleration increasing as the movement focuses in on the selectableobject. The distance moved by the person and the speed or accelerationmay further compound the rate of attraction or movement of theselectable object towards the selection object. In certain situations, anegative attractive force or gravitational effect may be used when it ismore desired that the selected objects move away from the user. Suchmotion of the objects would be opposite of that described above asattractive. As motion continues, the processing unit is able to betterdiscriminate between competing selectable objects and the one or onesmore closely aligned are pulled closer and separated, while othersrecede back to their original positions or are removed or fade. If themotion is directly toward a particular selectable object with acertainty above a threshold value, which has a certainty for examplegreater than 50%, then the selection and selectable objects merge andthe selectable object is simultaneously, synchronously or asynchronouslyselected and activated. Alternatively, the selectable object may beselected prior to merging with the selection object if the direction,angle, distance/displacement, duration, speed and/or acceleration of theselection object is such that the probability of the selectable objectis enough to cause selection, or if the movement is such that proximityto the activation area surrounding the selectable object is such thatthe threshold for selection, activation or both occurs. Motion continuesuntil the processing unit is able to determine that a selectable objecthas a selection threshold of greater than 50%, meaning that it morelikely than not the correct target object has been selected. In certainembodiments, the selection threshold will be at least 60%. In otherembodiments, the selection threshold will be at least 70%. In otherembodiments, the selection threshold will be at least 80%. In yet otherembodiments, the selection threshold will be at least 90%.Alternatively, the selection may be relative so that the selectioncertainty may be such that the certainty associated with one particularobject is higher by 50% or more than the certainties associated withother potentially selectable objects.

In certain embodiments, the selection object will actually appear on thedisplay screen, while in other embodiments, the selection object willexist only virtually in the processor software. For example, for motionsensors that require physical contact for activation such as touchscreens, the selection object may be displayed and/or virtual, withmotion on the screen used to determine which selectable objects from adefault collection of selectable objects will be moved toward aperceived or predefined location of a virtual section object or towardthe selection object in the case of a displayed selection object, whilea virtual object simply exists in software such as at a center of thedisplay or a default position to which selectable object are attracted,when the motion aligns with their locations on the default selection. Inthe case of motion sensors that have active zones such as cameras, IRsensors, sonic sensors, or other sensors capable of detecting motionwithin an active zone and creating an output representing that motion toa processing unit that is capable of determining direction, angle,distance/displacement, duration, speed and/or acceleration properties ofthe sensed or detected motion, the selection object is generally virtualand motion of one or more body parts of a user is used to attract aselectable object or a group of selectable objects to the location ofthe selection object and predictive software is used to narrow the groupof selectable objects and zero in on a particular selectable object,objects, objects and attributes, and/or attributes. In certainembodiments, the interface is activated from a sleep condition bymovement of a user or user body part into the active zone of the motionsensor or sensors associated with the interface.

Once activated, the feedback unit such as a display associated with theinterface displays or evidences in a user discernible manner a defaultset of selectable objects or a top level set of selectable objects. Theselectable objects may be clustered in related groups of similar objectsor evenly distributed about a centroid of attraction if no selectionobject is generated on the display or in or on another type of feedbackunit. If one motion sensor is sensitive to eye motion, then motion ofthe eyes will be used to attract and discriminate between potentialtarget objects on the feedback unit such as a display screen. If theinterface is an eye only interface, then eye motion is used to attractand discriminate selectable objects to the centroid, with selection andactivation occurring when a selection threshold is exceeded greater than50% confidence that one selectable object is more closely aligned withthe direction of motion than all other objects. The speed and/oracceleration of the motion along with the direction are further used toenhance discrimination by pulling potential target objects toward thecentroid quicker and increasing their size and/or increasing theirrelative separation. Proximity to the selectable object may also be usedto confirm the selection.

Alternatively, if the interface is an eye and other body part interface,then eye motion will act as the primary motion driver, with motion ofthe other body part acting as a confirmation of eye movement selections.Thus, if eye motion has narrowed the selectable objects to a group,which may or may not dynamically change the perspective of the user(zoom in/out, pan, tilt, roll, or any combination of changes) motion ofthe other body part may be used by the processing unit to furtherdiscriminate and/or select/activate a particular object or if aparticular object meets the threshold and is merging with the centroid,then motion of the object body part may be used to confirm or reject theselection regardless of the threshold confidence. In other embodiments,the motion sensor and processing unit may have a set of predeterminedactions that are invoked by a given structure of a body part or a givencombined motion of two or more body parts. For example, upon activation,if the motion sensor is capable of analyzing images, a hand holding updifferent number of figures from zero, a fist, to five, an open hand maycause the processing unit to display different base menus.

For example, a fist may cause the processing unit to display the toplevel menu, while a single finger may cause the processing unit todisplay a particular submenu. Once a particular set of selectableobjects is displayed, then motion attracts the target object, which issimultaneously, synchronously or asynchronously selected and activated.In other embodiments, confirmation may include a noised generated by theuses such as a word, a vocal noise, a predefined vocal noise, a clap, asnap, or other audio controlled sound generated by the user; in otherembodiments, confirmation may be visual, audio or haptic effects or acombination of such effects. In certain embodiments, the confirmationmay be dynamic, a variable sound, color, shape, feel, temperature,distortion, or any other effect or combination of thereof.

In other embodiments, the systems and apparatuses and implementinginterfaces and methods utilize body tracking, body part tracking, facetracking, head tracking, eye tracking, and any other tracking sensorthat detect movement of body parts over time and needs to include headworn devices (such as helmets and goggles) and devices that are notconnected to our bodies (such as looking through a simulator set ofglasses, screen, lenses or displays). We currently use a head-worndevice and use the IMU (MEMS) sensors and create a reticle (a bulls-eye)that represents the center-point of the gaze (is this the interactivecontrol construct?). We then use that reticle motion for controls andnavigation.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces of this disclosure include the steps ofsensing circular movement via a motion sensor, where the circularmovement is sufficient to activate a scroll wheel, scrolling through alist associated with the scroll wheel, where movement close to thecenter causes a faster scroll, while movement further from the centercauses a slower scroll and simultaneously, synchronously orasynchronously faster circular movement causes a faster scroll whileslower circular movement causes slower scroll. When the user stops thecircular motion, even for a very brief time, or changes direction suchthat it can be discerned to be no longer circular (such as moving in az-axis when the circular motion is in an xy plane) the list becomesstatic so that the user may move to a particular object, hold over aparticular object, or change motion direction at or near a particularobject. The whole wheel or a partial amount or portion of the wheel maybe displayed or just an arc may be displayed where scrolling moves upand down the arc. These actions cause the processing unit to select theparticular object, to simultaneously, synchronously or asynchronouslyselect and activate the particular object, or to simultaneously,synchronously or asynchronously select, activate, and control anattribute of the object. By beginning the circular motion again,anywhere on the screen, scrolling recommences immediately. Of course,scrolling could be through a list of values, or actually be controllingvalues as well, and all motions may be in 2D, 3D, and/or nD environmentsas well.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces of this disclosure include the steps ofdisplaying an arcuate menu layouts of selectable objects on a displayfield, sensing movement toward an object pulling the object toward theuser's location, user's movement, or center based on a direction, adistance/displacement, a duration, a speed and/or an acceleration of themovement, as the selected object moves toward user or the center,displaying subobjects appear distributed in an arcuate spaced apartconfiguration about the selected object. The apparatus, system andmethods can repeat the sensing and displaying operations. In all cases,singular or multiple subobjects or submenus may be displayed between theuser and the primary object, behind, below, or anywhere else as desiredfor the interaction effect.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces of this disclosure include the steps ofpredicting an object's selection based on the properties of the sensedmovement, where the properties includes direction, angle,distance/displacement, duration, speed, acceleration, changes thereof,or combinations thereof. For example, faster speed may increasepredictability, while slower speed may decrease predictability or viceversa. Alternatively, moving averages may be used to extrapolate thedesired object desired such as vector averages, linear and non-linearfunctions, including filters and multiple outputs form one or moresensors. Along with this is the “gravitational”, “electric” and/or“magnetic” attractive or repulsive effects utilized by the methods andsystems, whereby the selectable objects move towards the user orselection object and accelerates towards the user or selection object asthe user or selection object and selectable objects come closertogether. This may also occur by the user beginning motion towards aparticular selectable object, the particular selectable object begins toaccelerate towards the user or the selection object, and the user andthe selection object stops moving, but the particular selectable objectcontinues to accelerate towards the user or selection object. In thecertain embodiments, the opposite effect occurs as the user or selectionobjects moves away starting close to each other, the particularselectable object moves away quickly, but slows down its rate ofrepulsion as distance is increased, making a very smooth look. Indifferent uses, the particular selectable object might accelerate awayor return immediately to its original or predetermined position. In anyof these circumstances, a dynamic interaction is occurring between theuser or selection object and the particular selectable object(s), whereselecting and controlling, and deselecting and controlling can occur,including selecting and controlling or deselecting and controllingassociated submenus or subobjects and/or associated attributes,adjustable or invocable.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces of this disclosure include the steps ofdetecting at least one bio-kinetic characteristic of a user such as afingerprint, fingerprints, a palm print, retinal print, size, shape, andtexture of fingers, palm, eye(s), hand(s), face, etc. or at least oneEMF, acoustic, thermal or optical characteristic detectable by sonicsensors, thermal sensors, optical sensors, capacitive sensors, resistivesensors, or other sensor capable of detecting EMF fields, other dynamicwave form, or other characteristics, or combinations thereof emanatingfrom a user, including specific movements and measurements of movementsof body parts such as fingers or eyes that provide unique markers foreach individual, determining an identity of the user from thebio-kinetic characteristics, and sensing movement as set forth herein.In this way, the existing sensor for motion may also recognize the useruniquely, as well as the motion event associated with the user. Thisrecognition may be further enhanced by using two or more body parts orbio-kinetic characteristics (e.g., two fingers), and even further bybody parts performing a particular task such as being squeezed together,when the user enters in a sensor field. Other bio-kinetic and/orbiometric characteristics may also be used for unique useridentification such as skin characteristics and ratio to joint lengthand spacing. Further examples include the relationship between thefinger(s), hands or other body parts and the wave, acoustic, magnetic,EMF, or other interference pattern created by the body parts creates aunique constant and may be used as a unique digital signature. Forinstance, a finger in a 3D acoustic or EMF field would create uniquenull and peak points or a unique null and peak pattern, so the “noise”of interacting with a field may actually help to create uniqueidentifiers. This may be further discriminated by moving a certaindistance, where the motion may be uniquely identified by small tremors,variations, or the like, further magnified by interference patterns inthe noise. This type of unique identification may be used in touch andtouchless applications, but may be most apparent when using a touchlesssensor or an array of touchless sensors, where interference patterns(for example using acoustic sensors) may be present due to the size andshape of the hands or fingers, or the like. Further uniqueness may bedetermined by including motion as another unique variable, which mayhelp in security verification. Furthermore, by establishing a baseuser's bio-kinetic signature or authorization, slight variations perbio-kinetic transaction or event may be used to uniquely identify eachevent as well, so a user would be positively and uniquely identified toauthorize a merchant transaction, but the unique speed, angles, andvariations, even at a wave form and/or wave form noise level could beused to uniquely identify one transaction as compared to another.

In other embodiments, the methods for implementing the apparatuses,systems, and/or interfaces include the steps of sensing movement of afirst body part such as an eye, etc., tracking the first body partmovement until it stops, pauses or holds on an object, within an activezone of an object, or sufficient close to the active zone of the objectto insure select certainty to a threshold degree of certainty,preliminarily selecting the object, sensing movement of a second bodypart such as finger, hand, foot, etc., confirming the preliminaryselection and selecting the object. The selection may then cause theprocessing unit to invoke one of the command and control functionsincluding issuing a scroll function, a simultaneous, synchronous, orasynchronous select and scroll function, a simultaneous, synchronous, orasynchronous select and activate function, a simultaneous, synchronous,or asynchronous select, activate, and attribute adjustment function, ora combination thereof, and controlling attributes by further movement ofthe first or second body parts or activating the objects if the objectis subject to direct activation. These selection procedures may beinclude tracking eye motion/movement to initiate a scrolling functionthrough a list or over a list by the eye motion/movement or further eyemotion/movement toward the list or over the list and tracking movementof another body part to select a particular member of the list such asmoving a finger or hand, which motion/movement may also be used toconfirm the selection and selecting an object or a group of objects oran attribute or a group of attributes. In certain embodiments, if objectconfiguration is predetermined such that an object in the middle ofseveral objects, then the eye may move somewhere else, but hand motioncontinues to scroll or control attributes or combinations thereof,independent of the eyes. Hand and eyes may work together orindependently, or a combination in and out of the two. Thus, movementsmay be compound, sequential, simultaneous, synchronous or asynchronous,partially compound, compound in part, or combinations thereof. therequisite movement to scroll and select or select and confirm mayinvolve movement of two or more body parts, which are used to invoke oneof the command and control functions of this disclosure—selection andactivation, selection and attribute adjustment, selection and submenu orobject scrolling, or combination thereof. In certain embodiments, if theobject configuration is predetermined such that an object is in themiddle of several other objects, then once scrolling begins, thesystems, apparatuses, and/or interfaces may release the eyes so that theeyes may move somewhere else for performing other actions, while handmotion continues to scroll, select and/or control attributes orcombinations thereof, independent of eye movement. Thus, the systems,apparatuses, and/or interfaces may use hand and eye movement (movementof two or more body parts) in a cooperative manner, in a coupled manner,or in an independent manner to cause the systems, apparatuses, and/orinterfaces to control controllable object in different ways. Themovements may be independent, coupled, compound, sequential,simultaneous, concurrent, and/or complex, i.e., combining one or more ofindependent, coupled, compound, sequential, simultaneous, or concurrentmovements.

In other embodiments, the methods for implementing the apparatuses,systems, and/or interfaces include the steps of capturing a movement ofa user during a selection procedure or a plurality of selectionprocedures to produce a raw user movement dataset. The apparatuses,systems, interfaces, and/or methods also include the step of reducingthe raw user movement dataset to produce a refined user movementdataset, where the refinement may include reducing the movement to aplurality of linked vectors, to a fit curve, to a spline fit curve, toany other curve fitting format having reduced storage size, a reduceddata point collection, or to any other fitting format. The apparatuses,systems, interfaces, and/or methods also include the step of storing theraw user movement dataset or the refined user movement dataset in anappropriate database for storage, retrieval, and/or use by the systems,apparatuses, interfaces, and/or methods of this disclosure. Theapparatuses, systems, interfaces, and/or methods also include the stepof analyzing the raw user movement dataset and/or refined user movementdataset to produce a predictive tool for improving the prediction of auser's selection procedures using the motion-based system or to producea forensic tool for identifying the past behavior of the user or toproduce a training tool for training the user interface to improve userinteraction with the interface.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of sensing movement ofa plurality of body parts simultaneously, synchronously, orasynchronously or substantially simultaneously, synchronously, orasynchronously and converting the sensed movement into control functionsfor simultaneously, synchronously, or asynchronously controlling anobject, a plurality of objects, and/or attributes associated therewith.The apparatuses, systems, interfaces, and/or methods also includecontrolling an attribute or a plurality of attributes, or activating anobject or a plurality of objects, or any combination thereof. Forexample, placing a hand on a top of a domed surface for controlling anunmanned aerial vehicle (UAV), sensing movement of the hand on the dome,where a direction of movement correlates with a direction of flight,sensing changes in the movement on the top of the domed surface, wherethe changes correlate with changes in direction, angle,distance/displacement, duration, speed, or acceleration of functions,and simultaneously, synchronously, or asynchronously sensing movement ofone or more fingers, where movement of the fingers may control otherfeatures of the UAV such as pitch, yaw, roll, camera focusing, missilefiring, etc. with an independent finger(s) movement, while the hand,palm or other designated area of the hand is controlling the UAV, eitherthrough remaining stationary (continuing last known command) or whilethe hand is moving, accelerating, or changing distance, displacement (in2D or 3D or in a flexible or deformable medium), or direction ofacceleration. In certain embodiments where the display device is aflexible device such as a flexible screen or flexible dome, the movementmay also include deforming the surface of the flexible device, changinga pressure on the surface, inside the volume of the dome, or similarsurface and/or volumetric deformations. These deformations may be usedin conjunction with the other motions.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of populating a displayfield with displayed primary objects and hidden secondary objects, wherethe primary objects include menus, programs, applications, attributes,devices, etc. and secondary objects include submenus, attributes,preferences, etc. The methods and systems also include sensing movement,highlighting one or more primary objects most closely aligned with adirection of the movement, predicting a primary object based on themovement, and simultaneously, synchronously, or asynchronously: (a)selecting the primary object, (b) displaying secondary objects mostclosely aligned with the direction of motion in a spaced apartconfiguration, (c) pulling the primary and secondary objects toward acenter of the display field or to a pre-determined area of the displayfield, and/or (d) removing, fading, or making inactive the unselectedprimary and secondary objects until making active again.

Alternately, zones in between primary and/or secondary objects may actas activating areas or subroutines that would act the same as theobjects. For instance, if someone were to move in between two objects in2D (a watch or mobile device), 3D space (virtual reality environmentsand altered reality environments), nD space (n-dimensional space such asx, y, z, t space represented in VR or AR environments) objects in thebackground may be rotated to the front and the front objects may berotated towards the back, or to a different level.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of populating a displayfield with displayed primary objects and offset active fields associatedwith the displayed primary objects, where the primary objects includemenus, object lists, alphabetic characters, numeric characters, symbolcharacters, other text based characters. The methods and systems alsoinclude sensing movement, highlighting one or more primary objects mostclosely aligned with a direction of the movement, predicting a primaryobject based on the movement, context, and/or movement and context, andsimultaneously, synchronously, or asynchronously: (a) selecting theprimary object, (b) displaying secondary (tertiary or deeper) objectsmost closely aligned with the direction of motion/movement in a spacedapart configuration, (c) pulling the primary and secondary or deeperobjects toward a center of the display field or to a pre-determined areaof the display field, and/or (d) removing, making inactive, or fading orotherwise indicating a non-selection status of the unselected primary,secondary, and deeper level objects.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of sensing movement ofan eye and simultaneously, synchronously, asynchronously, orsequentially moving elements of a list within a fixed window or viewingpane of a display field or a display or an active object hidden orvisible through elements arranged in a 2D, 3D, and/or nD matrix withinthe display field, where eye movement anywhere, in any direction in adisplay field regardless of the arrangement of elements such as iconsmoves through the set of selectable objects. Of course the window may bemoved with the movement of the eye to accomplish the same scrollingthrough a set of lists or objects, or a different result may occur bythe use of both eye position in relation to a display or volume(perspective), as other motions occur, simultaneously, synchronously,asynchronously, or sequentially. Thus, scrolling does not have to be ina linear fashion, the intent is to select an object and/or attributeand/or other selectable items regardless of the manner of motion linear,arcuate, angular, circular, spiral, random, or the like. Once an objectof interest is to be selected, then selection is accomplished either bymovement of the eye in a different direction, holding the eye in placefor a period of time over an object, movement of a different body part,or any other movement or movement type that affects the selection of anobject, attribute, audio event, facial posture, and/or biometric orbio-kinetic event. These same steps may be used with body only or acombination of multiple body parts and eye or head gaze or movement.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of sensing movement ofan eye, selecting an object, an object attribute or both by moving theeye in a pre-described change of direction such that the change ofdirection would be known and be different than a random eye movement, ora movement associated with the scroll (scroll being defined by movingthe eye all over the screen or volume of objects with the intent tochoose). Of course the eye may be replaced by any body part or objectunder the control of a body part.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of sensing eye movementvia a motion sensor, selecting an object displayed in a display fieldwhen the eye pauses at an object for a dwell time sufficient for themotion sensor to detect the pause and simultaneously, synchronously,asynchronously, or sequentially activating the selected object,repeating the sensing and selecting until the object is eitheractivatable or an attribute capable of direct control. In certainembodiments, the methods also comprise predicting the object to beselected from characteristics of the movement and/or characteristics ofthe manner in which the user moves. In other embodiments, eye trackingusing gaze instead of motion for selection/control via eye focusing(dwell time or gaze time) on an object and a body motion (finger, hand,etc.) scrolls through an associated attribute list associated with theobject, or selects a submenu associated with the object. Eye gazeselects a submenu object and body motion confirms selection (selectiondoes not occur without body motion), so body motion actually affectsobject selection.

In other embodiments, eye tracking using motion for selection/controleye movement is used to select a first word in a sentence of a worddocument. Selection is confirmed by body motion of a finger (e.g., rightfinger) which holds the position. Eye movement is then tracked to thelast word in the sentence and another finger (e.g., the left finger)confirms selection. Selected sentence is highlighted due to secondmotion defining the boundary of selection. The same effect may be had bymoving the same finger towards the second eye position (the end of thesentence or word). Movement of one of the fingers towards the side ofthe monitor (movement is in different direction than the confirmationmove) sends a command to delete the sentence. Alternatively, movement ofeye to a different location, followed by both fingers moving generallytowards that location results in the sentence being copied to thelocation at which the eyes stopped. This may also be used in combinationwith a gesture or with combinations of motions and gestures such as eyemovement and other body movements concurrently multiple inputs at oncesuch as UAV controls described below.

In other embodiments, looking at the center of picture or article andthen moving one finger away from center of picture or center of bodyenlarges the picture or article (zoom in). Moving finger towards centerof picture makes picture smaller (zoom out). What is important tounderstand here is that an eye gaze point, a direction of gaze, or amotion of the eye provides a reference point for body motion andlocation to be compared. For instance, moving a body part (say a finger)a certain distance away from the center of a picture in a touch ortouchless, 2D or 3D environment (area or volume as well), may provide adifferent view. For example, if the eye(s) were looking at a centralpoint in an area, one view would appear, while if the eye(s) werelooking at an edge point in an area, a different view would appear. Therelative distance of the motion would change, and the relative directionmay change as well, and even a dynamic change involving both eye(s) andfinger, could provide yet another change of motion. For example, bylooking at the end of a stick and using the finger to move the other endof it, the pivot point would be the end the eyes were looking at. Bylooking at the middle of the stick, then using the finger to rotate theend, the stick would pivot around the middle. Each of these movement maybe used to control different attributes of a picture, screen, display,window, or volume of a 3D projection, etc. What now takes two fingersmay be replaced by one due to the eye(s) acting as the missing finger.

These concepts are useable to manipulate the view of pictures, images,3D data or higher dimensional data, 3D renderings, 3D buildingrenderings, 3D plant and facility renderings, or any other type of 3D orhigher dimensional (nD) pictures, images, or renderings. Thesemanipulations of displays, pictures, screens, etc. may also be performedwithout the coincidental use of the eye, but rather by using the motionof a finger or object under the control or a user, such as by movingfrom one lower corner of a bezel, screen, or frame (virtual or real)diagonally to the opposite upper corner to control one attribute, suchas zooming in, while moving from one upper corner diagonally to theother lower corner would perform a different function, for examplezooming out. This motion may be performed as a gesture, where theattribute change might occur in at predefined levels, or may becontrolled variably so the zoom in/out function may be a function oftime, space, and/or distance. By moving from one side or edge toanother, the same predefined level of change, or variable change mayoccur on the display, picture, frame, or the like. For example, a TVscreen displaying a picture and zoom-in may be performed by moving froma bottom left corner of the frame or bezel, or an identifiable region(even off the screen) to an upper right potion. As the user moves, thepicture is magnified (zoom-in). By starting in an upper right corner andmoving toward a lower left, the system causes the picture to be reducedin size (zoom-out) in a relational manner to the distance or speed theuser moves. If the user makes a quick diagonally downward movement fromone upper corner to the other lower corner, the picture may be reducedby 50% (for example). This eliminates the need for using two fingersthat is currently popular as a pinch/zoom function.

By the user moving from a right side of the frame or bezel or predefinedlocation towards a left side, an aspect ratio of the picture may bechanged so as to make the picture tall and skinny By moving from a topedge toward a bottom edge, the picture may cause the picture to appearshort and wide. By moving two fingers from one upper corner diagonallytowards a lower corner, or from side to side, a “cropping” function maybe used to select certain aspects of the picture.

By taking one finger and placing it near the edge of a picture, frame,or bezel, but not so near as to be identified as desiring to use a sizeor crop control, and moving in a rotational or circular direction, thepicture could be rotated variably, or if done in a quick gesturalmotion, the picture might rotate a predefined amount, for instance 90degrees left or right, depending on the direction of the motion.

By moving within a central area of a picture, the picture may be moved“panned” variably by a desired amount or panned a preset amount, say 50%of the frame, by making a gestural or dynamic motion in the direction ofa desired panning. Likewise, these same motions may be used in a 3Denvironment for simple manipulation of object attributes. These are notspecific motions using predefined pivot points as is currently used inCAD programs, but is rather a way of using the body (eyes or fingers forexample) in broad areas. These same motions may be applied to anydisplay, projected display or other similar device. In a mobile device,where many icons (objects) exist on one screen, where the icons includefolders of “nested” objects, by moving from one lower corner of thedevice or screen diagonally toward an upper corner, the display may zoomin, meaning the objects would appear magnified, but fewer would bedisplayed. By moving from an upper right corner diagonally downward, theicons would become smaller, and more could be seen on the same display.Moving in a circular motion near an edge of the display may causerotation of the icons, providing scrolling through lists and pages oficons. Moving from one edge to an opposite edge would change the aspectratio of the displayed objects, making the screen of icons appearshorter and wider, or taller and skinny, based on the direction moved.

In other embodiments, looking at a menu object then moving a finger awayfrom object or center of body opens up sub menus. If the objectrepresents a software program such as excel, moving away opens upspreadsheet fully or variably depending on how much movement is made(expanding spreadsheet window).

In other embodiments, instead of being a program accessed through anicon, the program may occupy part of a 3D space that the user interactswith or a field coupled to the program acting as a sensor for theprogram through which the user to interacts with the program. In otherembodiments, if object represents a software program such as Excel andseveral (say 4) spreadsheets are open at once, movement away from theobject shows 4 spread sheet icons. The effect is much like pullingcurtain away from a window to reveal the software programs that areopened. The software programs might be represented as “dynamic fields”,each program with its own color, say red for excel, blue for word, etc.The objects or aspects or attributes of each field may be manipulated byusing motion. For instance, if a center of the field is considered to bean origin of a volumetric space about the objects or value, moving at anexterior of the field cause a compound effect on the volume as a wholedue to having a greater x value, a greater y value, or a great zvalue—say the maximum value of the field is 5 (x, y, or z), moving at a5 point would be a multiplier effect of 5 compared to moving at a valueof 1 (x, y, or z) providing a gradient of values to interact with. Theinverse may also be used, where moving at a greater distance from theorigin may provide less of an effect on part or the whole of the fieldand corresponding values. Changes in color, shape, size, density, audiocharacteristics, or any combination of these and other forms ofrepresentation of values could occur, which may also help the user orusers to understand the effects of motion on the fields. These may bepreview panes of the spreadsheets or any other icons representing these.Moving back through each icon or moving the finger through each icon orpreview pane, then moving away from the icon or center of the bodyselects the open programs and expands them equally on the desktop, orlayers them on top of each other, etc. These actions may be combined,i.e., in AR/VR environments, where motion of the eyes and finger andanother hand (or body) may each or in combination have a predeterminedaxis or axes to display menus and control attributes or choices that maybe stationary or dynamic, and may interact with each other, so differentcombinations of eye, body and hand may provide the same results(redundantly), or different results based on the combination or sequenceof motions and holds, gazes, and even pose or posture in combinationwith these. Thus, motion in multiple axes may move in compound ways toprovide redundant or different effects, selection and attributecontrols.

In other embodiments, four word processor documents (or any program orweb pages) are open at once. Movement from bottom right of the screen totop left reveals the document at bottom right of page, effect looks likepulling curtain back. Moving from top right to bottom left reveals adifferent document. Moving from across the top, and circling back acrossthe bottom opens all, each in its quadrant, then moving through thedesired documents and creating circle through the objects links them alltogether and merges the documents into one document. As another example,the user opens three spreadsheets and dynamically combines or separatesthe spreadsheets merely via motions or movements, variably per amountand direction, angle, distance/displacement, and/or duration of themotion or movement. Again, the software or virtual objects are dynamicfields, where moving in one area of the field may have a differentresult than moving in another area, and the combining or moving throughthe fields causes a combining of the software programs, and may be donedynamically. Furthermore, using the eyes to help identify specificpoints in the fields (2D or 3D) would aid in defining the appropriatelayer or area of the software program (field) to be manipulated orinteracted with. Dynamic layers within these fields may be representedand interacted with spatially in this manner. Some or all the objectsmay be affected proportionately or in some manner by the movement of oneor more other objects in or near the field. Of course, the eyes may workin the same manner as a body part or in combination with other objectsor body parts. For instance, looking at the desired object and using atouch or touchless no-look user interface (UI) such as moving in acertain direction for selection, another for scrolling, etc. allows theuser to look and control with a combination of gaze and touch/touchlesscontrols. In all cases, contextual, environmental, prioritized, andweighted averages or densities and probabilities may affect theinteraction and aspect view of the field and the data or objectsassociated with the field(s). For instance, creating a graphicrepresentation of values and data points containing RNA, DNA, familyhistorical data, food consumption, exercise, etc., would interactdifferently if the user began interacting closer to the RNA zone than tothe food consumption zone, and the filed would react differently in partor throughout as the user moved some elements closer to others or in adifferent sequence from one are to another. This dynamic interaction andvisualization would be expressive of weighted values or combinations ofelements to reveal different outcomes.

In other embodiments, the eye selects (acts like a cursor hovering overan object and object may or may not respond, such as changing color toidentify it has been selected), then a motion or gesture of eye or adifferent body part confirms and disengages the eyes for furtherprocessing.

In other embodiments, the eye selects or tracks and a motion or movementor gesture of second body part causes a change in an attribute of thetracked object such as popping or destroying the object, zooming,changing the color of the object, etc. finger is still in control of theobject.

In other embodiments, eye selects, and when body motion and eye motionare used, working simultaneously, synchronously, asynchronously, orsequentially, a different result occurs compared to when eye motion isindependent of body motion, e.g., eye(s) tracks a bubble, finger movesto zoom, movement of the finger selects the bubble and now eye movementwill rotate the bubble based upon the point of gaze or change anattribute of the bubble, or the eye may gaze and select and/or control adifferent object while the finger continues selection and/or control ofthe first objector a sequential combination could occur, such as firstpointing with the finger, then gazing at a section of the bubble mayproduce a different result than looking first and then moving a finger;again a further difference may occur by using eyes, then a finger, thentwo fingers than would occur by using the same body parts in a differentorder.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of: controllinghelicopter with one hand on a domed interface, where several fingers andhand all move together and move separately. In this way, the wholemovement of the hand controls the movement of the helicopter inaltitude, direction, yaw, pitch and roll, while the fingers may alsomove simultaneously, synchronously, asynchronously, or sequentially tocontrol cameras, artillery, or other controls or attributes, or both.Thus, the systems, apparatuses and interfaces may process multiplemovement outputs from one or a plurality of motion sensorssimultaneously, congruently, or sequentially, where the movements may bedependent, partially dependent, partially coupled, fully coupled,partially independent or fully independently. The term dependent meansthat one movement is dominant and all other movements are dependent onthe dominant movement. For examples, in control of a UAV or traversing aVR/AR environment, the set of controllables may including altitude,direction, angle, distance/displacement, duration, speed, velocity,acceleration, yaw, pitch, roll, etc., where in certain circumstances,altitude may be the dominate controllable and all other are dependent onthe altitude being so that all other controllables are performed at adesignated altitude. The term partially dependent means that a set ofmovement outputs include a dominate output and the other member of theset are dependent on the dominant movement. For example considering thesame set of controllables, velocity and altitude may be independent andother sets tied to each one of them. The term partially coupled meansthat some of the movement outputs are coupled to each other so that theyact in a pre-defined or predetermine manner, while other areindependent. For example, considering the same controllables, altitude,direction, angle, distance/displacement, duration, velocity andacceleration may be coupled as the UAV is traveling a predefined path,while the other controllables are independently controllable. The termfully coupled means that all of the movement outputs are coupled to eachother so that they act in a pre-defined or predetermine manner. Forexample, all of the UAV sensors may all be coupled so that all of thesensors are tracking one specific target. The term partially independentmeans that some of the movement outputs are independent, while some areeither dependent or coupled. For example, all of the sensor may betracking one specific target, while the UAV positioning controls may allbe independently controlled. The term fully independent means that eachmovement output is processed independently of the other outputs. Thus,movement of multiple inputs may be simultaneously, synchronously,asynchronously, sequentially, congruently or independently utilized tocontrol objects and attributes thereof.

In certain embodiments, the perspective of the user as gravitationaleffects and object selections are made in 3D space. For instance, as wemove in 3D space towards subobjects, using our previously submittedgravitational and predictive effects, each selection may change theentire perspective of the user so the next choices are in the center ofview or in the best perspective. This may include rotational aspects ofperspective, the goal being to keep the required movement of the usersmall and as centered as possible in the interface real estate. This isreally showing the aspect, viewpoint or perspective of the user, and isrelative. Since we are saying the objects and fields may be moved, orsaying the user may move around the field, it is really a relative.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of sensing movement ofa button or knob with motion controls associated therewith, either ontop of or in 3D, 3 space, on sides (whatever the shape), predictingwhich gestures are called by direction, distance/displacement, angle,duration, and/or speed of motion (may be amendment togravitational/predictive application). By definition, a gesture has apose-movement-pose then lookup table, then command if values equalvalues in lookup table. We may start with a pose, and predict thegesture by beginning to move in the direction of the final pose. As wecontinue to move, we would be scrolling through a list of predictedgestures until we may find the most probable desired gesture, causingthe command of the gesture to be triggered before the gesture iscompleted. Predicted gestures could be dynamically shown in a list ofchoices and represented by objects or text or colors or by some othermeans in a display. As we continue to move, predicted end results ofgestures would be dynamically displayed and located in such a place thatonce the correct one appears, movement towards that object, representingthe correct gesture, would select and activate the gestural command. Inthis way, a gesture could be predicted and executed before the totalityof the gesture is completed, increasing speed and providing morevariables for the user.

For example, in a keyboard application, current software use shapes ofgestures to predict words. Goggle® uses zones of letters (a group ofletters), and combinations of zones (gestures) to predict words. Wewould use the same gesture-based system, except we be able to predictwhich zone the user is moving towards based upon one or more movementproperties, meaning we would not have to actually move into the zone tofinish the gesture, but moving towards the zone would select or bring upchoice bubbles, and moving towards the bubble would select that bubble.Once a word is chose, a menu of expanding option could show, so onecould create a sentence by moving through a sentence “tree”.

In another example, instead of using a gesture such as “a pinch” gestureto select something in a touchless environment, movement towards makingthat gesture would actually trigger the same command So instead ofhaving to actually touch the finger to the thumb, just moving the fingertowards the thumb would cause the same effect to occur. Most helpful incombination gestures where a finger pointing gesture is followed by apinching gesture to then move a virtual object. By predicting thegesture, after the point gesture, the beginning movement of the pinchgesture would be faster than having to finalize the pinching motion.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of: sensing movementvia a motion sensor within a display field displaying a list of lettersfrom an alphabet, predicting a letter or a group of letters based on themotion, if movement is aligned with a single letter, simultaneously,synchronously, asynchronously, or sequentially selecting the letter orsimultaneously, synchronously, asynchronously, or sequentially movingthe group of letter forward until a discrimination between letters inthe group is predictively certain and simultaneously, synchronously,asynchronously, or sequentially selecting the letter, sensing a changein a direction of motion, predicting a second letter or a second groupof letter based on the motion, if movement is aligned with a singleletter, simultaneously, synchronously, asynchronously, or sequentiallyselecting the letter or simultaneously, synchronously, asynchronously,or sequentially moving the group of letters forward until adiscrimination between letters in the group is predictively certain andsimultaneously, synchronously, asynchronously, or sequentially selectthe letter, either after the first letter selection or the second letterselection or both, display a list of potential letters, words, and/orphrases beginning with either the first letter or the second letter,selecting a word from the word list by movement of a second body partsimultaneously, synchronously, asynchronously or sequentially selectedthe word and resetting the original letter display, and repeating thesteps until a message is completed.

Thus, the current design selects a letter simply by changing one or moremotion properties at or near a letter. A faster process would be to usemovement toward a letter, then changing one or more motion propertiesbefore reaching the letter and moving towards a next letter and changingone or more motion properties again before getting to the next lettermay better predict words, and might change the first letter selection.Selection bubbles may appear and be changing while moving, so one ormore movement properties may be used to predict the word, notnecessarily having to move over the exact letter or very close to it,though moving over the exact letter may be a positive selection of thatletter and this effect could be better verified by a slight pausing orslowing down of movement. (Of course, this may be combined with currentbutton like actions or lift-off events or touch-up events, and more thanone finger or hand may be used, both simultaneously, synchronously,asynchronously, or sequentially to provide the spelling and typingactions.) This is most effective in a touchless environment whererelative motion can be leveraged to predict words on a keyboard ratherthan the actual distance required to move from key to key. The distancefrom a projected keyboard and movement of finger using angles of motionto predict letters. Predictive word bubbles may be selected with a zmovement. Move below the letters of a keyboard to select, or shape theletter buttons in such a way that they extend downward (like a teardrop) so actual letters can be seen while selecting instead of coveringthe letters (the touch or active zones are offset from the actual keys.This may also be used with predictive motions to create a very fastkeyboard where relative motions are used to predict keys and words whilemore easily being able to see the key letters. Bubbles may also appearabove or besides the keys, or around them, including in an arcuate orradial fashion to further select predicted results by moving towards thesuggested words.

In other embodiments, the methods for implementing the systems,apparatuses, and/or interfaces include the steps of: maintaining allsoftware applications in an instant on configuration—on, but inactive,resident, but not active, so that once selected the application which ismerely dormant, is fully activate instantaneously (or may be describedas a different focus of the object), sensing movement via a motionsensor with a display field including application objects distributed onthe display in a spaced apart configuration. In certain embodiments, amaximally spaced apart configuration is invoked so that the movementresults in a fast predict selection of an application object, pulling anapplication object or a group of application objects toward a center ofthe display field, if movement is aligned with a single application,simultaneously, synchronously, asynchronously, or sequentially selectand instant on the application, or continue monitoring the movementuntil a discrimination between application objects is predictivelycertain and simultaneously, synchronously, asynchronously, orsequentially selecting and activating the application object.

Thus, the industry must begin to start looking at everything as alwayson and what is on is always interactive, and may have different levelsof interactivity. For instance, software should be an interactive field.Excel and word should be interactive fields where motion through themmay combine or select areas, which correspond to cells and texts beingintertwined with the motion. Excel sheets should be part of the same 3Dfield, not separate pages, and should have depth so their aspects can becombined in volume. The software desktop experience needs a depth wherethe desktop is the cover of a volume, and rolling back the desktop fromdifferent corners reveals different programs that are active and havedifferent colors, such as word being revealed when moving from bottomright to top left and being a blue field, excel being revealed whenmoving from top left to bottom right and being red; moving right to leftlifts desktop cover and reveals all applications in volume, eachapplication with its own field and color in 3D space.

In other embodiments, the systems, apparatuses, and/or interfacesinclude an active screen area having a delete or backspace region. Whenthe user moves the active object (cursor) toward the delete or backspaceregion, then the selected objects will be released one at a time or ingroups or completely depending on attributes of movement toward thedelete of backspace region. Thus, if the movement is slow and steady,then the selected objects are released one at a time. If the movement isfast, then multiple selected objects are released. Thus, the delete orbackspace region is variable. For example, if the active display regionrepresents a cell phone dialing pad (with the number distributed in anydesired configuration from a traditional grid configuration to a arcuateconfiguration about the active object, or in any other desirableconfiguration), when by moving the active object toward the delete orbackspace region, numbers will be removed from the number, which may bedisplayed in a number display region of the display. Alternatively,touching the backspace region would back up one letter; moving fromright to left in the backspace region would delete (backspace) acorresponding amount of letters based on the distance (and/or speed) ofthe movement, The deletion could occur when the motion is stopped,paused, or a lift off event is detected. Alternatively, a swiping motion(jerk, or fast acceleration) could result in the deletion (backspace)the entire word. All these may or may not require a lift off event, butthe motion dictates the amount deleted or released objects such asletters, numbers, or other types of objects. The same is true with thedelete key, except the direction would be forward instead of backwards.Lastly, the same may be true in a radial menu (or linear or spatial),where the initial motion towards an object or on an object, or in a zoneassociated with an object, that has a variable attribute. The motionassociated with or towards that object may provide immediate control.

In other embodiments, the systems, apparatuses, and/or interfacesutilize eye movement to select and body part movement is used to confirmor activate the selection. Thus, eye movement is used as the selectivemovement, while the object remains in the selected state, then the bodypart movement confirms the selection and activates the selected object.Thus, specifically stated the eye or eyes look in a different directionor area, and the last selected object would remain selected until adifferent object is selected by motion of the eyes or body, or until atime-out deselects the object. An object may be also selected by an eyegaze, and this selection would continue even when the eye or eyes are nolonger looking at the object. The object would remain selected unless adifferent selectable object is looked at, or unless a timeout deselectsthe object occurs.

In all of the embodiments set forth above, the motion or movement mayalso comprise lift off events, where a finger or other body part orparts are in direct contract with a touch sensitive feedback device suchas a touch screen, then the acceptable forms of motion or movement willcomprise touching the screen, moving on or across the screen, liftingoff from the screen (lift off events), holding still on the screen at aparticular location, holding still after first contact, holding stillafter scroll commencement, holding still after attribute adjustment tocontinue an particular adjustment, holding still for different periodsof time, moving fast or slow, moving fast or slow or different periodsof time, accelerating or decelerating, accelerating or decelerating fordifferent periods of time, changing direction, changingdistance/displacement, changing duration, changing speed, changingvelocity, changing acceleration, changing direction for differentperiods of time, changing speed for different periods of time, changingvelocity for different periods of time, changing acceleration fordifferent periods of time, or any combinations of these motions may beused by the systems and methods to invoke command and control over realworld or virtual world controllable objects using on the motion only.Lift off or other events could “freeze” the state of menu, object orattribute selection, or combination of these, until another event occursto move to a different event or state, or a time-out function resets thesystem or application to a pre-configured state or location. A virtuallift off could accomplish the same effect in a VR, AR or realenvironment, by moving in a different direction or designated directionwith no physical lift off event. Of course, if certain objects that areinvoked by the motion sensitive processing of the systems and methods ofthis disclosure require hard select protocols mouse clicks, fingertouches, etc., the invoked object's internal function will not beaugmented by the systems or methods of this disclosure unless theinvoked object permits or supports system integration. In place ofphysical or virtual lift offs or confirmations could be sounds, colorsor contextual or environmental triggers.

The systems, apparatuses, and/or interfaces and the methods forimplementing them are disclosed herein where command functions forselection and/or control of real and/or virtual objects may be generatedbased on a change in velocity at constant direction, a change indirection at constant velocity, a change in both direction and velocity,a change in a rate of velocity, or a change in a rate of acceleration ora change in distance/displacement and time. Once detected by an detectoror sensor, these changes may be used by a processing unit to issuecommands for controlling real and/or virtual objects. A selection orcombination scroll, selection, and attribute selection may occur uponthe first movement. Such motion may be associated with doors opening andclosing in any direction, golf swings, virtual or real world games,light moving ahead of a runner, but staying with a walker, or any othermotion having compound properties such as direction, angle,distance/displacement, duration, velocity, acceleration, and changes inany one or all of these primary properties; thus, direction, angle,distance/displacement, duration, velocity, and acceleration may beconsidered primary motion properties, while changes in these primaryproperties may be considered secondary motion or movement properties.The system may then be capable of differentially handling of primary andsecondary motion or movement properties. Thus, the primary propertiesmay cause primary functions to be issued, while secondary properties maycause primary function to be issued, but may also cause the modificationof primary function and/or secondary functions to be issued. Forexample, if a primary function comprises a predetermined selectionformat, the secondary motion or movement properties may expand orcontract the selection format.

In another example of this primary/secondary format for causing thesystem to generate command functions may involve an object display.Thus, by moving the object in a direction away from the user's eyes orusing changes in angle, distance/displacement, velocity, acceleration/ortime, the state of the display may change, such as from a graphic to acombination graphic and text, to a text display only, while moving sideto side or moving a finger or eyes from side to side may scroll thedisplayed objects or change the font or graphic size, while moving thehead to a different position in space might reveal or control attributesor submenus of the object. Thus, these changes in motions may bediscrete, compounded, or include changes in velocity, acceleration andrates of these changes to provide different results for the user. Theseexamples illustrate two concepts: 1) the ability to have compoundmotions which provide different results that the motions separately orsequentially, and (2) the ability to change states or attributes, suchas graphics to text solely or in combination with single or compoundmotions, or with multiple inputs, such as verbal, touch, facialexpressions, or bio-kinetically, all working together to give differentresults, or to provide the same results in different ways.

It must be recognized that the present disclosure while based on the useof sensed velocity, acceleration, and changes and rates of changes inthese properties to effect control of real world objects and/or virtualobjects, the present disclosure may also use other properties of thesensed motion in combination with sensed velocity, acceleration, andchanges in these properties to effect control of real world and/orvirtual objects, where the other properties include direction, angle,distance/displacement, duration and change in direction, angle,distance/displacement, duration of motion, where the motion has aconstant velocity. For example, if the motion sensor(s) senses velocity,acceleration, changes in velocity, changes in acceleration, and/orcombinations thereof that is used for primary control of the objects viamotion of a primary sensed human, animal, part thereof, real worldobject under the control of a human or animal, or robots under controlof the human or animal, then sensing motion of a second body part may beused to confirm primary selection protocols or may be used to fine tunethe selected command and control function. Thus, if the selection is fora group of objects, then the secondary motion or movement properties maybe used to differentially control object attributes to achieve a desiredfinal state of the objects.

It must be recognized that the present disclosure while based on the useof sensed velocity, acceleration, and changes and rates of changes inthese properties to effect control of real world objects and/or virtualobjects, the present disclosure may also use other properties of thesensed motion in combination with sensed velocity, acceleration, andchanges in these properties to effect control of real world and/orvirtual objects, where the other properties include direction, angle,distance/displacement, duration and change in direction, angle,distance/displacement, duration of motion, where the motion has aconstant velocity. For example, if the motion sensor(s) senses velocity,acceleration, changes in velocity, changes in acceleration, and/orcombinations thereof that is used for primary control of the objects viamotion of a primary sensed human, animal, part thereof, real worldobject under the control of a human or animal, or robots under controlof the human or animal, then sensing motion of a second body part may beused to confirm primary selection protocols or may be used to fine tunethe selected command and control function. Thus, if the selection is fora group of objects, then the secondary motion or movement properties maybe used to differentially control object attributes to achieve a desiredfinal state of the objects.

For example, suppose the apparatuses of this disclosure control lightingin a building. There are banks of lights on or in all four walls(recessed or mounted) and on or in the ceiling (recessed or mounted).The user has already selected and activated lights from a selection menuusing motion to activate the apparatus and motion to select and activatethe lights from a list of selectable menu items such as sound system,lights, cameras, video system, etc. Now that lights has been selectedfrom the menu, movement to the right would select and activate thelights on the right wall. Movement straight down would turn all of thelights of the right wall down dim the lights. Movement straight up wouldturn all of the lights on the right wall up brighten. The velocity ofthe movement down or up would control the rate that the lights weredimmed or brighten. Stopping movement would stop the adjustment orremoving the body, body part or object under the user control within themotion sensing area would stop the adjustment.

For even more sophisticated control using motion or movement properties,the user may move within the motion sensor active area to map out adownward concave arc, which would cause the lights on the right wall todim proportionally to the arc distance from the lights. Thus, the rightlights would be more dimmed in the center of the wall and less dimmedtoward the ends of the wall.

Alternatively, if the movement was convex downward, then the light woulddim with the center being dimmed the least and the ends the most.Concave up and convex up would cause differential brightening of thelights in accord with the nature of the curve.

Now, the apparatus may also use the velocity of the movement of themapping out the concave or convex movement to further change the dimmingor brightening of the lights. Using velocity, starting off slowly andincreasing speed in a downward motion would cause the lights on the wallto be dimmed more as the motion moved down. Thus, the lights at one endof the wall would be dimmed less than the lights at the other end of thewall.

Now, suppose that the motion is a S-shape, then the light would bedimmed or brightened in a S-shaped configuration. Again, velocity may beused to change the amount of dimming or brightening in different lightssimply by changing the velocity of movement. Thus, by slowing themovement, those lights would be dimmed or brightened less than when themovement is speed up. By changing the rate of velocity accelerationfurther refinements of the lighting configuration may be obtained.

Now suppose that all the lights in the room have been selected, thencircular or spiral motion would permit the user to adjust all of thelights, with direction, angle, distance/displacement, duration, velocityand acceleration properties being used to dim and/or brighten all thelights in accord with the movement relative to the lights in the room.For the ceiling lights, the circular motion may move up or down in the zdirection to affect the luminosity of the ceiling lights. Thus, throughthe sensing of motion or movement within an active sensor zone area andespecially volume, a user can use simple or complex motion todifferentially control large numbers of devices simultaneously,synchronously, asynchronously or sequentially. By scrolling through thearea (pointing the finger at each light) and stopping motion at eachlight desired it would be selected, then moving in a different directionwould allow for attribute of only the selected lights. The same wouldhold for virtual objects in a 2D or 3D (VR/AR) environment. Thus, a useris able to select groups of objects that may represent real or virtualobjects and once the group is selected, movement of the user may adjustall object and/or device attribute collectively. This feature isespecially useful when the interface is associated with a large numberof object, subobjects, and/or devices and the user wants to selectedgroups of these objects, subobjects, and/or devices so that they may becontrolled collectively. Thus, the user may navigate through theobjects, subobjects and/or devices and select any number of them bymoving to each object pausing so that the system recognizes to add theobject to the group. Once the group is defined, the user would be ableto save the group as a predefined group or just leave it as a temporarygroup. Regardless, the group would not act as a single object for theremainder of the session. The group may be deselected by moving outsideof the active field of sensor, sensors, and/or sensor arrays.

This differential control through the use of sensed complex motionpermits a user to nearly instantaneously change lighting configurations,sound configurations, TV configurations, or any configuration of systemshaving a plurality of devices being simultaneously, synchronously,asynchronously, or sequentially controlled or of a single system havinga plurality of objects or attributes capable of simultaneous,synchronous, asynchronous, or sequential control. For examples, in acomputer game including large numbers of virtual objects such as troops,tanks, airplanes, etc., sensed complex motion would permit the user toquickly deploy, redeploy, rearrangement, manipulated and generallyquickly reconfigure all controllable objects and/or attributes by simplyconforming the movement of the objects to the movement of the usersensed by the motion detector. This same differential device and/orobject control would find utility in military and law enforcement, wherecommand personnel by motion or movement within a sensing zone of amotion sensor quickly deploy, redeploy, rearrangement, manipulated andgenerally quickly reconfigure all assets to address a rapidly changingsituation.

The all of these scenarios set forth above are designed to illustratethe control of a large number of devices using properties and/orcharacteristics of the sensed motion including, without limitation,relative distance of the motion for each object (real like a person in aroom using his/her hand as the object for which motion is being sensedor virtual representations of the objects in a virtual or rendered roomon a display apparatus), direction of motion, distance/displacement ofmotion, duration of motion, speed of motion, acceleration of motion,changes an any of these properties, rates of changes in any of theseproperties, or mixtures and combinations thereof to control a singlecontrollable attribute of the object such as lights. However, thesystems, apparatuses, and methods of this disclosure are also capable ofusing motion or movement properties and/or characteristics to controltwo, three, or more attributes of an object. Additionally, the systems,apparatuses, and methods of this disclosure are also capable of usingmotion or movement properties and/or characteristics from a plurality ofmoving objects within a motion sensing zone to control differentattributes of a collection of objects. For example, if the lights in theabove figures are capable of color as well as brighten, then the motionor movement properties and/or characteristic may be used tosimultaneously, synchronously, asynchronously or sequentially changecolor and intensity of the lights or one sensed motion could controlintensity, while another sensed motion could control color. For example,if an artist wanted to paint a picture on a computer generated canvas,then motion or movement properties and/or characteristic would allow theartist to control the pixel properties of each pixel on the displayusing the properties of the sensed motion from one, two, three, etc.sensed motions. Thus, the systems, apparatuses, and methods of thisdisclosure are capable of converting the motion or movement propertiesassociated with each and every object being controlled based on theinstantaneous properties values as the motion traverse the object inreal space or virtual space.

The systems, apparatuses and methods of this disclosure activate uponmotion being sensed by one or more motion sensors. This sensed motionthen activates the systems and apparatuses causing the systems andapparatuses to process the motion and its properties activating aselection object and a plurality of selectable objects. Once activated,the motion or movement properties cause movement of the selection objectaccordingly, which will cause a pre-selected object or a group ofpre-selected objects, to move toward the selection object, where thepre-selected object or the group of pre-selected objects are theselectable object(s) that are most closely aligned with the direction ofmotion, which may be evidenced by the user feedback units bycorresponding motion of the selection object.

Another aspect of the systems or apparatuses of this disclosure is thatthe faster the selection object moves toward the pre-selected object orthe group of preselected objects, the faster the pre-selected object orthe group of preselected objects move toward the selection object.Another aspect of the systems or apparatuses of this disclosure is thatas the pre-selected object or the group of pre-selected objects movetoward the selection object, the pre-selected object or the group ofpre-selected objects may increase in size, change color, becomehighlighted, provide other forms of feedback, or a combination thereof.Another aspect of the systems or apparatuses of this disclosure is thatmovement away from the objects or groups of objects may result in theobjects moving away at a greater or accelerated speed from the selectionobject(s). Another aspect of the systems or apparatuses of thisdisclosure is that as motion continues, the motion will start todiscriminate between members of the group of pre-selected object(s)until the motion results in the selection of a single selectable objector a coupled group of selectable objects.

Once the selection object and the target selectable object touch, activeareas surrounding the objection touch, a threshold distance between theobject is achieved, or a probability of selection exceeds an activationthreshold, the target object is selected and non-selected displayobjects are removed from the display, change color or shape, or fadeaway or any such attribute so as to recognize them as not selected. Thesystems or apparatuses of this disclosure may center the selected objectin a center of the user feedback unit or center the selected object ator near a location where the motion was first sensed. The selectedobject may be in a corner of a display on the side the thumb is on whenusing a phone, and the next level menu is displayed slightly furtheraway, from the selected object, possibly arcuately, so the next motionis close to the first, usually working the user back and forth in thegeneral area of the center of the display. If the object is anexecutable object such as taking a photo, turning on a device, etc.,then the execution is simultaneous, synchronous, asynchronous orsequential with selection. If the object is a submenu, sublist or listof attributes associated with the selected object, then the submenumembers, sublist members or attributes are displayed on the screen in aspaced apart format. The same procedure used to select the selectedobject is then used to select a member of the submenu, sublist orattribute list. Thus, the interfaces have a gravity like or anti-gravitylike action on display objects. As the selection object(s) moves, itattracts an object or objects in alignment with the direction of theselection object's motion pulling those object(s) toward it and maysimultaneously, synchronously, asynchronously or sequentially repelnon-selected items away or indicate non-selection in any other manner soas to discriminate between selected and non-selected objects As motioncontinues, the pull increases on the object most aligned with thedirection of motion, further accelerating the object toward theselection object until they touch or merge or reach a threshold distancedetermined as an activation threshold. The touch or merge or thresholdvalue being reached causes the processing unit to select and activatethe object(s).

Additionally, the sensed motion may be one or more motions detected byone or more movements within the active zones of the motion sensor(s)giving rise to multiple sensed motions and multiple command functionthat may be invoked simultaneously, synchronously, asynchronously orsequentially. The sensors may be arrayed to form sensor arrays. If theobject is an executable object such as taking a photo, turning on adevice, etc., then the execution is simultaneous, synchronous,asynchronous or sequential with selection. If the object is a submenu,sublist or list of attributes associated with the selected object, thenthe submenu members, sublist members or attributes are displayed on thescreen in a spaced apart format. The same procedure used to select theselected object is then used to select a member of the submenu, sublistor attribute list. Thus, the interfaces have a gravity like action ondisplay objects. As the selection object moves, it attracts an object orobjects in alignment with the direction of the selection object's motionpulling those object toward it. As motion continues, the pull increaseson the object most aligned with the direction of motion, furtheraccelerating the object toward the selection object until they touch ormerge or reach a threshold distance determined as an activationthreshold to make a selection. The touch, merge or threshold eventcauses the processing unit to select and activate the object.

The sensed motion may result not only in activation of the systems,apparatuses, and/or interfaces, but may be result in select, attributecontrol, activation, actuation, scroll or combination thereof.

In other embodiments, the systems, apparatuses, and/or interfaces mayutilize different unit feedback formats including haptic or tactileoutputs, audio outputs, any other cognizable feedback formats toindicate or evident user interaction with the systems, apparatuses,and/or interfaces. For example, if the user moving through radial zones,different objects may produce different buzzes or sounds, and theintensity or pitch may change while moving in that zone to indicatewhether the object is in front of or behind the user.

Compound motions may also be used so as to provide different controlfunction than the motions made separately or sequentially. This includescombination attributes and changes of both state and attribute, such astilting the device to see graphics, graphics and text or text, alongwith changing scale based on the state of the objects, while providingother controls simultaneously, synchronously, asynchronously,sequentially or independently, such as scrolling, zooming in/out, orselecting while changing state. These features may also be used tocontrol chemicals being added to a vessel, while simultaneously,synchronously, asynchronously or sequentially controlling the amount.These features may also be used to change between operating systems suchas between Windows® 8 and Windows® 7 with a tilt while moving icons orscrolling through programs at the same time.

Audible or other communication medium may be used to confirm objectselection or in conjunction with motion so as to provide desiredcommands (multimodal) or to provide the same control commands indifferent ways.

In other embodiments, the systems, apparatuses, and/or interfaces andthe methods for implementing them may also include artificialintelligence (AI) components (e.g., any AI component including orcontaining, but is not limited to, a Machine Learning (ML), DeepLearning (DL), Neural Network (NN) and any other multi-parameter,self-learning component) that learn from a user's interaction with thesystems, apparatuses, and/or interfaces including movementcharacteristics, environmental characteristics (e.g., motion sensortypes, processing unit types, or other environment properties),controllable object environmental characteristics, historicalcharacteristics, etc. to improve the performance of the systems,apparatuses, and/or interfaces and to improve prediction capability ofthe systems, apparatuses, and/or interfaces to anticipate useractivities, or to modify the content or processes. The AI components maybe utilized to affect modification to the training routines to enhancethe training routine by including actions taken by a majority (greaterthan 50%) of individuals utilizing the training routine. For example, ifthe AI component determines that additional specific information andtraining exercises are required for a given task or task component, theAI routine may cause the systems, interfaces, elements, or apparatusesand the methods implementing them to update the task or task component,content, and/or processes by incorporating new content/information aboutthe task or task component or updating aspects of the task or taskcomponent to improve task or task component completion and proficiency.Moreover, the AI analysis may be used to update an entire trainingroutine or any task or component of the training routine. Again, the AIroutine will generally update the training routine if a majority oftrainers and/or trainees respond to the training routine, component,task, or task component in a similar fashion or manner. The AIcomponents may also be used to analyze trainer interaction fordetermining trainer proclivities for performing the routine, components,tasks and/or task components and/or trainee interaction for determiningtrainee proclivities for performing the routine, components, tasksand/or task components and modify the training routine for specifictypes of trainees like determining whether a student is an auditory orvisual or mixed learner. Thus, the systems, interfaces, or apparatusesand the methods implementing them may include trainer and/or traineetype specific implementations of the training routine. The systems,interfaces, or apparatuses and the methods implementing them would thenhave a preliminary interaction procedure, wherein a new trainee or a newtrainer would perform some preliminary tasks designed to determine thetrainers and/or trainees learning proclivities and select a typespecific implementation that would best accommodate the new trainerand/or trainee improving the trainee's ability to complete a trainingroutine with increase proficiency. This same AI system or elements maybe used to approximate, replicate or predict motion events, gestures,and any function(s) of scrolling, selecting, activating, or attributecontrol or any combination of these. For instance, using AI, the systemcould determine a certain vector is being selected and the change inmotion rate that normally would occur along that vector can be recreatedprobabilistically through parameters and algorithms. These may them becombined with real motion and other real parameters to provide theeffect/outcomes of artificial motion events, where the real motion didnot need to occur. Artificial Vectors and vector attributes can becreated and used in the same way.

Embodiments of this disclosure further relate to systems for selectingand activating virtual or real objects and their controllable attributesincluding at least one motion sensor having an active sensing zone, atleast one processing unit, at least one power supply unit, and oneobject or a plurality of objects under the control of the processingunits. The sensors, processing units, and power supply units are inelectrical communication with each other. The motion sensors sensemotion including motion or movement properties within the active zones,generate at least one output signal, and send the output signals to theprocessing units. The processing units convert the output signals intoat least one command function. The command functions include (1) a startfunction, (2) a scroll function, (3) a select function, (4) an attributefunction, (5) an attribute control function, (6) a simultaneous,synchronous, asynchronous or sequential control function including: (7)a select and scroll function, (8) a select, scroll and activatefunction, (9) a select, scroll, activate, and attribute controlfunction, (10) a select and activate function, (11) a select andattribute control function, (12) a select, activate, and attributecontrol function, or (13) combinations thereof, or (14) combinationsthereof. The start functions activate at least one selection or cursorobject and a plurality of selectable objects upon first sensing motionby the motion sensors and selectable objects aligned with the motiondirection move toward the selection object or become differentiated fromnon-aligned selectable objects and motion continues until a targetselectable object or a plurality of target selectable objects arediscriminated from non-target selectable objects resulting in activationof the target object or objects. The motion or movement propertiesinclude a touch, a lift off, a direction, a distance/displacement, aduration, a velocity, an acceleration, a change in direction, a changein distance/displacement, a change in duration, a change in velocity, achange in acceleration, a rate of change of direction, a rate of changeof distance/displacement, a rate of change of duration, a rate of changeof velocity, a rate of change of acceleration, stops, holds, timedholds, or mixtures and combinations thereof. The objects comprise realworld objects, virtual objects and mixtures or combinations thereof,where the real world objects include physical, mechanical,electro-mechanical, magnetic, electro-magnetic, electrical, orelectronic devices or any other real world device that can be controlledby a processing unit and the virtual objects include any constructgenerated in a virtual world or by a computer and displayed by a displaydevice and that are capable of being controlled by a processing unit.The attributes comprise activatable, executable and/or adjustableattributes associated with the objects. The changes in motion ormovement properties are changes discernible by the motion sensors sensoroutputs, and/or the processing units.

In certain embodiments, the start functions further activate the userfeedback units and the selection objects and the selectable objects arediscernible via the motion sensors in response to movement of an animal,human, robot, robotic system, part or parts thereof, or combinationsthereof within the motion sensor active zones. In other embodiments, thesystem further includes at least on user feedback unit, at least onebattery backup unit, communication hardware and software, at least oneremote control unit, or mixtures and combinations thereof, where thesensors, processing units, power supply units, the user feedback units,the battery backup units, the remote control units are in electricalcommunication with each other. In other embodiments, faster motioncauses a faster movement of the target object or objects toward theselection object or causes a greater differentiation of the targetobject or object from the non-target object or objects. In otherembodiments, if the activated objects or objects have subobjects and/orattributes associated therewith, then as the objects move toward theselection object, the subobjects and/or attributes appear and becomemore discernible as object selection becomes more certain. In otherembodiments, once the target object or objects have been selected, thenfurther motion within the active zones of the motion sensors causesselectable subobjects or selectable attributes aligned with the motiondirection to move towards the selection object(s) or becomedifferentiated from non-aligned selectable subobjects or selectableattributes and motion continues until a target selectable subobject orattribute or a plurality of target selectable objects and/or attributesare discriminated from non-target selectable subobjects and/orattributes resulting in activation of the target subobject, attribute,subobjects, or attributes. In other embodiments, the motion sensor isselected from the group consisting of digital cameras, optical scanners,optical roller ball devices, touch pads, inductive pads, capacitivepads, holographic devices, laser tracking devices, thermal devices,acoustic devices, any other device capable of sensing motion, arrays ofmotion sensors, and mixtures or combinations thereof. In otherembodiments, the objects include lighting devices, cameras, ovens,dishwashers, stoves, sound systems, display systems, alarm systems,control systems, medical devices, robots, robotic control systems, hotand cold water supply devices, air conditioning systems, heatingsystems, ventilation systems, air handling systems, computers andcomputer systems, chemical plant control systems, computer operatingsystems, virtual reality systems, augmented reality systems, graphicssystems, business software systems, word processor systems, internetbrowsers, accounting systems, military systems, control systems, othersoftware systems, programs, routines, objects and/or elements, remotecontrol systems, or mixtures and combinations thereof. In otherembodiments, if the timed hold is brief, then the processing unit causesan attribute to be adjusted to a preset level. In other embodiments, ifthe timed hold is continued, then the processing unit causes anattribute to undergo a high value/low value cycle that ends when thehold is removed. In other embodiments, the timed hold causes anattribute value to change so that (1) if the attribute is at its maximumvalue, the timed hold causes the attribute value to decrease at apredetermined rate, until the timed hold is removed, (2) if theattribute value is at its minimum value, then the timed hold causes theattribute value to increase at a predetermined rate, until the timedhold is removed, (3) if the attribute value is not the maximum orminimum value, then the timed hold causes randomly selects the rate anddirection of attribute value change or changes the attribute to allowmaximum control, or (4) the timed hold causes a continuous change in theattribute value in a direction of the initial motion until the timedhold is removed. In other embodiments, the motion sensors sense a secondmotion including second motion or movement properties within the activezones, generate at least one output signal, and send the output signalsto the processing units, and the processing units convert the outputsignals into a confirmation command confirming the selection or at leastone second command function for controlling different objects ordifferent object attributes. In other embodiments, the motion sensorssense motions including motion or movement properties of two or moreanimals, humans, robots, or parts thereof, or objects under the controlof humans, animals, and/or robots within the active zones, generateoutput signals corresponding to the motions, and send the output signalsto the processing units, and the processing units convert the outputsignals into command function or confirmation commands or combinationsthereof implemented simultaneously, synchronously, asynchronously orsequentially, where the start functions activate a plurality ofselection or cursor objects and a plurality of selectable objects uponfirst sensing motion by the motion sensor and selectable objects alignedwith the motion directions move toward the selection objects or becomedifferentiated from non-aligned selectable objects and the motionscontinue until target selectable objects or pluralities of targetselectable objects are discriminated from non-target selectable objectsresulting in activation of the target objects and the confirmationcommands confirm the selections.

The inventors have found that systems and methods implemented on aprocessing unit such as a computer may be constructed that permit thecreation of dynamic environments for object and/or attribute display,manipulation, differentiation, and/or interaction, where the systemsinclude one processing unit or a plurality of processing units, onemotion sensor or a plurality of motion sensors, one user interface or aplurality of user interfaces and dynamic environment software forgenerating, displaying, and manipulating the dynamic environments andthe objects and/or attributes included therein. The dynamic environmentsare produced via user interaction with the sensor(s), which are inelectronic communication with the processing unit(s), and comprise a setof objects and associated attributes displayed on the user interface(s)so that the objects and/or attributes are differentiated one from theother. The differentiation may evidence priority, directionality,content, type, activation procedures, activation parameters, controlfeatures, other properties that are associated with the objects and/orattributes or combinations thereof. The differentiation and distributionof the objects and/or attributes may change based on user interactionwith the motion sensors and/or locations of the motion sensors, where atleast one motion sensor or sensor output is associated with a mobile orstationary device or where at least one motion sensor or sensor outputis associated with a mobile device and at least one motion sensor orsensor output is associated with a stationary device, and mixtures orcombinations thereof. Of course, these same procedures may be used withobjects and/or attributes at any level of drill down.

In certain embodiments, the systems and methods of this disclosureactivation of the system causes a plurality of selectable objects to bedisplayed on a display device of a user interface associated with thesystems. The selectable objects may be represent: (1) objects that maydirectly invoked, (2) objects that have a single attribute, (3) objectsthat have a plurality of attributes, (4) objects that are lists or menusthat may include sublists or submenus, (5) any other selectable item, or(6) mixtures and combinations thereof. The objects may represent virtualor real objects. Virtual objects may be any object that represents aninternal software component. Real object may be executable programs orsoftware application or may be real world devices that may be controlledby the systems and/or methods. The displayed selectable objects may be adefault set of selectable objects, pre-defined set of selectableobjects, or a dynamically generated set of selectable objects, generatedbased on locations of the sensors associated with mobile devices and themotion sensors associated with stationary devices. The systems andmethods permit the selectable objects to interact with the userdynamically so that object motion within the environments bettercorrelates with the user ability to interact with the objects. The userinteractions include, but are not limited to: (a) object discriminationbased on sensed motion, (b) object selection base on sensed motion, (c)menu drill down based on sensed motion, (d) menu drill up based onsensed motion, (e) object selection and activation based on sensedmotion and on the nature of the selectable object, (f)scroll/selection/activation based on sensed motion and on the nature ofthe selectable object, and (g) any combination of the afore listedinteractions associated with a collection of linked objects, where thelinking may be pre-defined, based on user gained interaction knowledge,or dynamically generated based on the user, sensor locations, and thenature of the sensed motion. The systems and methods may also associateone or a plurality of object differentiation properties with thedisplayed selectable objects, where the nature of the differentiationfor each object may be predefined, defined based on user gainedinteraction knowledge, or dynamically generated based on the user,sensor locations, and/or the nature of the sensed motion. Thedifferentiation properties include, but are not limited to: color; colorshading; spectral attributes associated with the shading; highlighting;flashing; rate of flashing; flickering; rate of flickering; shape; size;movement of the objects such as oscillation, side to side motion, up anddown motion, in and out motion, circular motion, elliptical motion,zooming in and out, etc.; rate of motion; pulsating; rate of pulsating;visual texture; touch texture; sounds such as tones, squeals, beeps,chirps, music, etc.; changes of the sounds; rate of changes in thesounds; any user discernible object differentiation properties, or anymixture and combination thereof. The differentiation may signify to theuser a sense of direction, object priority, object sensitivity, etc.,all helpful to the user for dynamic differentiation of selectableobjects displayed on the display derived from the user, sensed motion,and/or the location of the mobile and stationary sensors.

For example, one displayed object may pulsate (slight zooming in andout, or expanding and contracting) at a first rate, while anotherdisplayed object may pulsate a second rate, where the first and secondrates may be the same or different, and a faster pulsation rate may beassociated with a sense of urgency relative to objects having a slowerrate of pulsation. These rates may change in a pre-defined manner, amanner based on knowledge of the user, or dynamically based on the user,sensor locations, and/or the nature of the sensed motion.

In another example, a set of objects may slightly move to the rightfaster than they move back to the left, indicating that the user shouldapproach the objects from the right, instead from another direction.

In certain embodiments, a main object may have one or a plurality ofsub-objects moving (constant or variable rate and/or direction) aroundor near the main object, indicating the nature of the sub-objects. Inthis case, sub-objects revolving around the main object may representthat they need to be interacted with in a dynamic, motion-based way,whereas the main object may be interacted with in a static manner suchas a vocal command, hitting a button, clicking, or by any othernon-dynamic or static interaction.

In other embodiments, a main object may have a certain color, such asblue, and its associated sub-objects have shades of blue, especiallywhere the sub-objects dynamically transition from blue to off-blue orblue-green or other related colors, displaying they come from theprimary blue object, whereas a red Object next to the blue one mighthave sub-objects that transition to orange, while a sub-object thattransitions to purple might represent it is a sub-set of blue and redand can be accessed through either.

In other embodiments, the objects or sub-objects may fade in or out,representing changes of state based on a time period that the userinteracts with them. By fading out, the systems may be notifying theuser that the program or application (e.g., water flow in a building)will be entering a sleep or interruption state. The rate of the fade outmay indicate how quickly the program or application transitions into asleep state and how quickly they reactivate. A fade-in might relay theinformation that the object will automatically initiate over a giventime automatically vs. manually.

In other embodiments, an array of objects, such as the screen ofapplications on a mobile device, the objects pulsing might representprograms that are active, whereas the objects that are static mightrepresent programs that are inactive. Programs that are pulsing at aslower rate might represent programs running occasionally in thebackground. Of course, other dynamic indicators, such as changes incolor, intensity, translucency, size, shape, or any recognizableattribute, may be used to relay information to the user.

Another example of the operation of the systems and methods of thisdisclosure may be in a medical context. In such a case, the objectsdisplayed on the user interface may be an array of sensors active in anoperating room including, but not limited to, oxygen sensors, blood flowsensors, pulse rate sensors, heart beat rate, blood pressure sensors,brain activity sensors, etc. The different dynamic changes in color,shape, size, sound, and/or movement of the objects may represent dataassociated with the sensors, providing multiple points of information ina simple, compounded way to the user. If color represented oxygen level,size represented pressure, and dynamic movement of the objectrepresented heartbeat, one object could represent a great deal ofinformation to the user.

The characteristics of associated sub-objects seen simultaneously,synchronously, asynchronously or sequentially after the primary objectsare selected and may likewise provide much more information than justletting the user know more information exists in this case, the primaryobject would be labeled with the corresponding body position and thesub-object representing oxygen level past and current data might bepulsing or intensifying dynamically in color, while the blood pressuresub-object might be slightly growing larger or smaller with eachheartbeat, representing minimal change in blood pressure, and theheartbeat might be represented by the object rotating clockwise (CW),then counter clockwise (CAW) with each heartbeat.

In another example, one object (or word in a word document) swappingplaces with another might represent the need to change the word toprovide better grammar for a sentence. Spelling changes might berepresented by pulsing words, and words that are acceptable, but have abetter common spelling might be represented by words that pulse at aslower rate. Dynamic changes of color might also be associated with thewords or other characteristics to draw attention to the user and givesecondary information at the same time, such as which words that mightbe too high or too low of a grade level for the reader in school books.

Thus, any combination of dynamic characteristics may be used to providemore information to the user than a static form of information, and maybe used in conjunction with the static information characteristic.

In certain embodiments, objects (such as application icons) may haveseveral possible states and display states. An object may be in anunselected state, a present state (available for selection but with noprobability of being selected yet), a pre-selected (now probable, butnot meeting a threshold criteria for being selected), a selected state(selected but not opened or having an execute command yet issued), or anactuated state (selected and having an attribute executed (i.e., on (vs.off), variable control ready to change based on moving up or down, or asubmenu is displayed and ready to be selected). If the object is in agroup of objects, as the user moves towards that group, the zone and/orthe group of objects may display or present a different characteristicthat represents they are ready to be selected; this may be identified asa pre-selected state. In each state, the objects may display differentcharacteristics to convey information to the user, such as change ofshape, size, color, sound, smell, feel, pulse rate, different dynamicdirectional animations, etc. For instance, before a user touches amobile device (one with a touch sensor), the objects may be in anunselected state, displaying no attribute other than the common staticdisplay currently employed. Once a user touches the screen, the itemsthat need attention might change in color (present, but no differentprobability of being selected than any others). As the user begins tomove in the direction of an object desired, the more likely objects maybegin to display differently, such as increasing in size, or beginpulsing, and as the probability increases, the pulse rate may increase,but objects in more urgent need of attention may pulse differently oreven faster than others in the same group or zone—pre-selected. Once thecorrect object(s) is selected, it may show and even different state,such as displaying subobjects, changing color, or making a sound, but itstill may not be open or actuated yet. If the attribute is volumecontrol, it may be selected, but would not control volume until it isactuated by moving up or down, adjusting the volume. Of course, objectsin an unselected state may show dynamic characteristics (pulsing forexample) as well to convey information to the user, such as activity orpriority. In this way, it may have a dynamic characteristic while in astatic state.

In another example, for applications in the corner of a mobile device,when, head or eye gaze is directed towards that zone or objects, theymay be in an unselected, preselected, or selected but not actuatedstate, and they may demonstrate dynamic indicators/attributes to conveyintent, attributes, sub-attributes, or mixed or combination content orattributes with changing environments. They may display differently atany state, or only at one particular state (such as selected), and thismay be a preset value, or something dynamic, such as contextual orenvironmental factors. An example of this last dynamic characteristicindicator would be in a vehicle or virtual reality display where thesong playlist would cause a pulsing effect on preferred songs, butdifferent songs would pulse differently when another occupant or playerenters the environment, indicating the suggested objects would changedue a combination of user preferences, and the dynamic displaycharacteristics of all or some of the objects would change to indicate acombination preferential selections).

The dynamic environment systems of this disclosure may also be used invirtual reality systems and/or augmented reality systems so that playersor users of these virtual reality systems and/or augmented realitysystems through motion and motion properties are able to select, target,and/or deselect features, menus, objects, constructs, constructions,user attributes, weapons, personal attributes, personal features, anyother selectable or user definable features or attributes of the virtualspace or augmented reality space. Thus, as a user enters first enters avirtual reality space or augment reality space, all of the selectable ordefinable features and/or attributes of the space would be displayedabout the user in any desired form 2D and/or 3D semicircular orhemispherical array with user at center, 2D and/or 3D circular orspherical array with user at center, 2D and/or 3D matrix array with userat center or off-center, any other 2D and/or 3D display of features andattributes, or mixtures and combinations thereof. As the user moves abody part associated with the motion detectors used to interface withthe space (visual eye tracking sensors, hand part sensors gloves or thelike, body sensors body suits, or other sensors), the sensed motions andmotion properties such as direction, angle, distance/displacement,duration, speed, acceleration, and/or changes in any of these motionproperties cause features and/or attributes to display differently basedon state and information to display to the user, and may move toward theuser based on the motion and motion or movement properties of the objectand/or the user, while the other features and/or attributes stay staticor move away from the user. An example of this is to move towards aparticular tree in a group of trees in a game. As the user looks towarda particular tree, the tree might shake while the others sway gently, asthe user moves toward the tree, the tree may begin to move towards theuser at a faster rate, if has a special prize associated with it, or ata slower rate in no prize. If the special prize is a one of a kindattribute, the tree may change color or size at it moves towards theuser and the user is moving towards the tree. Once the tree is selectedvia a threshold event, it may change shape into the prize it held, andthen the start to act like that prize when it is selected by the usermoving the hand towards a designated area of the object enough toactuate. These different attributes or characteristics are part of adynamic environment where the speed, direction, angle,distance/displacement, duration, state, display characteristics andattributes are affected by motion of the user and object, or anycombination of these. In another example, where it is desired to chooseone object, as the motion or motion properties of user(s), object(s) orboth continue, the features and/or attributes are further of user,objects or both are discriminated, and the target features and/orattributes may move closer. Once the target is fully differentiated,then all subfeatures and/or subobjects may become visible. As motioncontinues, features and/or attributes and/or subfeatures and/orsubobjects are selected and the user gains the characteristics orfeatures the user desires in the space. All of the displayed featuresand/or attributes and/or subfeatures and/or subobjects may also includehighlighting features such as sound (chirping, beeping, singing, etc.),vibration, back and forth movement, up and down movement, circularmovement, etc.

Embodiments of this disclosure relate broadly to computing devices,comprising at least one sensor or sensor output configured to capturedata including user data, motion data, environment data, temporal data,contextual data, and/or mixtures and combinations thereof. The computingdevice also includes at least one processing unit configured, based onthe captured data, to generate at least one command function. Thecommand functions comprise: (1) a single control function including (a)a start function, (b) a scroll function, (c) a select function, (d) anattribute function, (e) an activate function, or (f) mixtures andcombinations thereof. The command functions also comprise: (2) asimultaneous, synchronous, asynchronous, or sequential control functionincluding (a) a combination of two or more of the functions (1a-1e), (b)a combination of three or more of the functions (1a-1e), (c) acombination of four or more of the functions (1a-1e), (d) mixtures andcombinations thereof. The command functions may also comprise (3)mixtures and combinations of any of the above functions. In certainembodiments, the at least one sensor comprises touch pads, touchlesspads, inductive sensors, capacitive sensors, optical sensors, acousticsensors, thermal sensors, optoacoustic sensors, electromagnetic field(EMF) sensors, wave or waveform sensors, strain gauges, accelerometers,any other sensor that senses movement or changes in movement, ormixtures and combinations thereof. In other embodiments, a first controlfunction is a single control function. In other embodiments, a firstcontrol function is a single control function and a second function is asimultaneous, synchronous, asynchronous or sequential control function.In other embodiments, a first control function is a simultaneous,synchronous, asynchronous or sequential control function. In otherembodiments, a first control function is a simultaneous, synchronous,asynchronous or sequential control function and a second function is asimultaneous, synchronous, asynchronous or sequential control function.In other embodiments, a plurality of single and simultaneous,synchronous, asynchronous or sequential control functions are actuatedby user determined motion.

Embodiments of this disclosure relate broadly to computer implementedmethods, comprising under the control of a processing unit configuredwith executable instructions, receiving data from at least one sensorconfigured to capture the data, where the captured data includes userdata, motion data, environment data, temporal data, contextual data, ormixtures and combinations thereof. The methods also comprise processingthe captured data to determine a type or types of the captured data;analyzing the type or types of the captured data; and invoking a controlfunction corresponding to the analyzed data. The control functionscomprise: (1) a single control function including: (a) a start function,(b) a scroll function, (c) a select function, (d) an attribute function,(e) an activate function, or (f) mixtures and combinations thereof, or(2) a simultaneous, synchronous, asynchronous or sequential controlfunction including: (a) a combination of two or more of the functions(1a-1e), (b) a combination of three or more of the functions (1a-1e),(c) a combination of four or more of the functions (1a-1e), (d) mixturesand combinations thereof, or (3) mixtures and combinations thereof. Incertain embodiments, the at least one sensor comprises touch pads,touchless pads, inductive sensors, capacitive sensors, optical sensors,acoustic sensors, thermal sensors, optoacoustic sensors, electromagneticfield (EMF) sensors, strain gauges, accelerometers, any other sensorthat senses movement or changes in movement, or mixtures andcombinations thereof. In other embodiments, a first control function isa single control function. In other embodiments, a first controlfunction is a single control function and a second function is asimultaneous, synchronous, asynchronous or sequential control function.In other embodiments, a first control function is a simultaneous,synchronous, asynchronous or sequential control function. In otherembodiments, a first control function is a simultaneous, synchronous,asynchronous or sequential control function and a second function is asimultaneous, synchronous, asynchronous or sequential control function.In other embodiments, a plurality of single and simultaneous,synchronous, asynchronous or sequential control functions are actuatedby user determined motion.

Embodiments of this disclosure relate broadly to non-transitory computerreadable storage media storing one or more sequences of instructionsthat, when executed by one or more processing units, cause a computingsystem to: (a) receive data from at least one sensor configured tocapture the data, where the captured data includes user data, motiondata, environment data, temporal data, contextual data, or mixtures andcombinations thereof; (b) process the captured data to determine a typeor types of the captured data; (c) analyze the type or types of thecaptured data; and (d) invoke a control function corresponding to theanalyzed data. The control functions comprise (1) a single controlfunction including: (a) a start function, (b) a scroll function, (c) aselect function, (d) an attribute function, (e) an activate function, or(f) mixtures and combinations thereof, or (2) a simultaneous,synchronous, asynchronous or sequential control function including: (a)a combination of two or more of the functions (1a-1e), (b) a combinationof three or more of the functions (1a-1e), (c) a combination of four ormore of the functions (1a-1e), (d) mixtures and combinations thereof, or(3) mixtures and combinations thereof. In certain embodiments, the atleast one sensor comprises touch pads, touchless pads, inductivesensors, capacitive sensors, optical sensors, acoustic sensors, thermalsensors, optoacoustic sensors, electromagnetic field (EMF) sensors,strain gauges, accelerometers, any other sensor that senses movement orchanges in movement, or mixtures and combinations thereof. In otherembodiments, a first control function is a single control function. Inother embodiments, a first control function is a single control functionand a second function is a simultaneous, synchronous, asynchronous orsequential control function. In other embodiments, a first controlfunction is a simultaneous, synchronous, asynchronous or sequentialcontrol function. In other embodiments, a first control function is asimultaneous, synchronous, asynchronous or sequential control functionand a second function is a simultaneous, synchronous, asynchronous orsequential control function. In other embodiments, a plurality of singleand simultaneous, synchronous, asynchronous or sequential controlfunctions are actuated by user determined motion.

Embodiments of this disclosure relate broadly to computer-implementedsystems comprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, and a memory; a computer program including instructionsexecutable by the digital processing device to create a gesture-basednavigation environment. The environment comprises a software moduleconfigured to receive input data from a motion sensor, the input datarepresenting navigational gestures of a user; a software moduleconfigured to present one or more primary menu items; and a softwaremodule configured to present a plurality of secondary menu items inresponse to receipt of input data representing a navigational gesture ofthe user indicating selection of a primary menu item, the secondary menuitems arranged in a curvilinear orientation about the selected primarymenu item. The environment operates such that in response to receipt ofinput data representing a navigational gesture of the user comprisingmotion substantially parallel to the curvilinear orientation, theplurality of secondary menu items scrolls about the curvilinearorientation; in response to receipt of input data representing anavigational gesture of the user substantially perpendicular to thecurvilinear orientation, an intended secondary menu item in line withthe direction of the navigational gesture is scaled and moved oppositeto the direction of the navigational gesture to facilitate user access.In certain embodiments, the processing device or unit is a smart watchand the motion sensor is a touchscreen display.

Embodiments of this disclosure relate broadly to non-transitorycomputer-readable storage media encoded with a computer programincluding instructions executable by a processor to create agesture-based or motion-based navigation environment comprising: asoftware module configured to receive input data from a motion sensor,the input data representing navigational movement of a user; a softwaremodule configured to present one or more primary menu items; and asoftware module configured to present a plurality of secondary menuitems in response to receipt of input data representing a navigationalgesture or movement of the user indicating selection of a primary menuitem, the secondary menu items arranged in a curvilinear orientationabout the selected primary menu item. The environment operates such thatin response to receipt of input data representing a navigational gestureor movement of the user comprising motion substantially parallel to thecurvilinear orientation, the plurality of secondary menu items scrollsabout the curvilinear orientation; and in response to receipt of inputdata representing a navigational gesture or movement of the usersubstantially perpendicular to the curvilinear orientation, an intendedsecondary menu item in line with the direction of the navigationalgesture or movement is scaled and moved opposite to the direction of thenavigational gesture or movement to facilitate user access. In certainembodiments, the processor is a smart watch or wearable device and themotion sensor is a touchscreen display.

Embodiments of this disclosure relate broadly to systems for selectingand activating virtual or real objects and their controllable attributescomprising: at least one motion sensor having an active sensing zone, atleast one processing unit, at least one power supply unit, one object ora plurality of objects under the control of the processing units. Thesensors, processing units, and power supply units are in electricalcommunication with each other. The motion sensors sense motion includingmotion or movement properties within the active zones, generate at leastone output signal, and send the output signals to the processing units.The processing units convert the output signals into at least onecommand function. The command functions comprise: (1) a start function,(2) a scroll function, (3) a select function, (4) an attribute function,(5) an attribute control function, (6) a simultaneous, synchronous,asynchronous, or sequential control function, and/or (7) combinationsthereof. The simultaneous, synchronous, asynchronous, or sequentialcontrol functions include: (1) a select and scroll function, (2) aselect, scroll and activate function, (3) a select, scroll, activate,and attribute control function, (4) a select and activate function, (5)a select and attribute control function, (6) a select, activate, andattribute control function, or (7) combinations thereof. The startfunctions activate at least one selection or cursor object and aplurality of selectable objects upon first sensing motion by the motionsensors and selectable objects aligned with the motion direction movetoward the selection object or become differentiated from non-alignedselectable objects and motion continues until a target selectable objector a plurality of target selectable objects are discriminated fromnon-target selectable objects resulting in activation of the targetobject or objects. The motion or movement properties include a touch, alift off, an angle, a direction, a distance/displacement, a duration, avelocity, an acceleration, a change in direction, a change indistance/displacement, a change in duration, a change in velocity, achange in acceleration, a rate of change of direction, a rate of changeof distance/displacement, a rate of change of duration, a rate of changeof velocity, a rate of change of acceleration, stops, holds, timedholds, and/or mixtures and combinations thereof. The objects comprisereal world objects, virtual objects and mixtures or combinationsthereof, where the real world objects include physical, mechanical,electro-mechanical, magnetic, electro-magnetic, electrical, orelectronic devices or any other real world device that can be controlledby a processing unit and the virtual objects include any constructgenerated in a virtual world or by a computer and displayed by a displaydevice and that are capable of being controlled by a processing unit.The attributes comprise selectable, activatable, executable and/oradjustable attributes associated with the objects. The changes in motionor movement properties are changes discernible by the motion sensorsand/or the processing units. In certain embodiments, the start functionsfurther activate the user feedback units and the selection objects andthe selectable objects are discernible via the motion sensors inresponse to movement of an animal, human, robot, robotic system, part orparts thereof, or combinations thereof within the motion sensor activezones. In other embodiments, the systems further comprise: at least onuser feedback unit, at least one battery backup unit, communicationhardware and software, at least one remote control unit, or mixtures andcombinations thereof. The sensors, processing units, power supply units,the user feedback units, the battery backup units, the remote controlunits are in electrical communication with each other. In otherembodiments, the systems further comprise: at least one battery backupunit, where the battery backup units are in electrical communicationwith the other hardware and units. In other embodiments, faster motioncauses a faster movement of the target object or objects toward theselection object or objects or causes a greater differentiation of thetarget object or objects from non-target object or objects. In otherembodiments, the non-target object or objects move away from theselection object as the target object or objects move toward theselection object or objects to aid in object differentiation. In otherembodiments, the target objects and/or the non-target objects aredisplayed in list, group, or array forms and are either partially orwholly visible or partially or wholly invisible. In other embodiments,if the activated object or objects have subobjects and/or attributesassociated therewith, then as the object or objects move toward theselection object, the subobjects and/or attributes appear and becomemore discernible as the target object or objects becomes more certain.In other embodiments, the target subobjects and/or the non-targetsubobjects are displayed in list, group, or array forms and are eitherpartially or wholly visible or partially or wholly invisible. In otherembodiments, once the target object or objects have been selected, thenfurther motion within the active zones of the motion sensors causesselectable subobjects or selectable attributes aligned with the motiondirection to move towards, away and/or at an angle to the selectionobject(s) or become differentiated from non-aligned selectablesubobjects or selectable attributes and motion continues until a targetselectable subobject or attribute or a plurality of target selectableobjects and/or attributes are discriminated from non-target selectablesubobjects and/or attributes resulting in activation of the targetsubobject, attribute, subobjects, or attributes. In other embodiments,the motion sensor is selected from the group consisting of digitalcameras, optical scanners, optical roller ball devices, touch pads,inductive pads, capacitive pads, holographic devices, laser trackingdevices, thermal devices, acoustic devices, any other device capable ofsensing motion, arrays of motion sensors, waveform sensors, sensingdevices, and mixtures or combinations thereof. In other embodiments, theobjects include lighting devices, cameras, ovens, dishwashers, stoves,sound systems, display systems, alarm systems, control systems, medicaldevices, robots, robotic control systems, hot and cold water supplydevices, air conditioning systems, heating systems, ventilation systems,air handling systems, computers and computer systems, chemical plantcontrol systems, computer operating systems, software elements, softwareroutines, graphics systems, business software systems, word processorsystems, internet browsers, accounting systems, military softwaresystems, virtual reality systems, augmented reality systems, mixedreality systems, software controlling mixed, augmented, and/or virtualreality environments, biometric systems, neurological systems, dronecontrols, and systems, software control systems, other software systems,programs, routines, objects and/or elements, remote control systems,and/or mixtures and combinations thereof. In other embodiments, if thetimed hold is brief, then the processing unit causes an attribute to beadjusted to a preset level. In other embodiments, if the timed hold iscontinued, then the processing unit causes an attribute to undergo ahigh value/low value cycle that ends when the hold is removed. In otherembodiments, the timed hold causes an attribute value to change so that(1) if the attribute is at its maximum value, the timed hold causes theattribute value to decrease at a predetermined rate, until the timedhold is removed, (2) if the attribute value is at its minimum value,then the timed hold causes the attribute value to increase at apredetermined rate, until the timed hold is removed, (3) if theattribute value is not the maximum or minimum value, then the timed holdcauses randomly selects the rate and direction of attribute value changeor changes the attribute to allow maximum control, or (4) the timed holdcauses a continuous change in the attribute value in a direction of theinitial motion until the timed hold is removed. In other embodiments,the motion sensors sense a second motion including second motion ormovement properties within the active zones, generate at least oneoutput signal, and send the output signals to the processing units, andthe processing units convert the output signals into a confirmationcommand confirming the selection or at least one second command functionfor controlling different objects or different object attributes. Inother embodiments, the motion sensors sense motions including motion ormovement properties of two or more animals, humans, robots, or partsthereof, or objects under the control of humans, animals, and/or robotswithin the activate zones, generate output signals corresponding to themotions, and send the output signals to the processing units, and theprocessing units convert the output signals into command function orconfirmation commands or combinations thereof implementedsimultaneously, synchronously, asynchronously, or sequentially, wherethe start functions activate a plurality of selection or cursor objectsand a plurality of selectable objects upon first sensing motion by themotion sensor and selectable objects aligned (co-linear with or the sameangle) with the motion directions move toward the selection objects orbecome differentiated from non-aligned selectable objects and themotions continue until target selectable objects or pluralities oftarget selectable objects are discriminated from non-target selectableobjects resulting in activation of the target objects and theconfirmation commands confirm the selections.

Embodiments of this disclosure relate broadly to methods for controllingobjects comprising: sensing motion including motion or movementproperties within an active sensing zone of at least one motion sensor,where the motion or movement properties include a direction, an angle, adistance/displacement, a duration, a velocity, an acceleration, a changein direction, a change in distance/displacement, a change in duration, achange in velocity, a change in acceleration, a rate of change ofdirection, a rate of change of distance/displacement, a rate of changeof duration, a rate of change of velocity, a rate of change ofacceleration, stops, holds, timed holds, and/or mixtures andcombinations thereof, producing an output signal or a plurality ofoutput signals corresponding to the sensed motion, converting the outputsignal or signals via a processing unit in communication with the motionsensors into a command function or a plurality of command functions. Thecommand functions comprise: (1) a start function, (2) a scroll function,(3) a select function, (4) an attribute function, (5) an attributecontrol function, (6) a simultaneous, synchronous, asynchronous, orsequential control function including: (7) a select and scroll function,(8) a select, scroll and activate function, (9) a select, scroll,activate, and attribute control function, (10) a select and activatefunction, (11) a select and attribute control function, (12) a select,activate, and attribute control function, or (13) combinations thereof,or (14) combinations thereof. The methods also include processing thecommand function or the command functions simultaneously, synchronously,asynchronously, or sequentially, where the start functions activate atleast one selection or cursor object and a plurality of selectableobjects upon first sensing motion by the motion sensor and selectableobjects aligned with the motion direction move toward the selectionobject or become differentiated from non-aligned selectable objects andmotion continues until a target selectable object or a plurality oftarget selectable objects are discriminated from non-target selectableobjects resulting in activation of the target object or objects, wherethe motion or movement properties include a touch, a lift off, adirection, an angle, a distance/displacement, a duration, a velocity, anacceleration, a change in direction, a change in angle, a change indistance/displacement, a change in duration, a change in velocity, achange in acceleration, a rate of change of direction, a rate of changeof distance/displacement, a rate of change of angle, a rate of change ofduration, a rate of change of velocity, a rate of change ofacceleration, stops, holds, timed holds, and/or mixtures andcombinations thereof. The objects comprise real world objects, virtualobjects, attributes associated therewith, and/or mixtures andcombinations thereof, where the real world objects include physical,mechanical, electro-mechanical, magnetic, electro-magnetic, electrical,or electronic devices or any other real world device that can becontrolled by a processing unit and the virtual objects include anyconstruct generated in a virtual world or by a computer and displayed bya display device and that are capable of being controlled by aprocessing unit. The attributes comprise activatable, executable and/oradjustable attributes associated with the objects.

The changes in motion or movement properties are changes discernible bythe motion sensors and/or the processing units. In certain embodiments,the motion sensor is selected from the group consisting of digitalcameras, optical scanners, optical roller ball devices, touch pads,inductive pads, capacitive pads, holographic devices, laser trackingdevices, thermal devices, acoustic devices, any other device capable ofsensing motion, arrays of motion sensors, and mixtures or combinationsthereof. In other embodiments, the objects include lighting devices,cameras, ovens, dishwashers, stoves, sound systems, display systems,alarm systems, control systems, medical devices, robots, robotic controlsystems, hot and cold water supply devices, air conditioning systems,heating systems, ventilation systems, air handling systems, computersand computer systems, chemical plant control systems, computer operatingsystems, systems, graphics systems, business software systems, wordprocessor systems, internet browsers, accounting systems, militarysystems, virtual reality systems, augmented reality systems, controlsystems, other software systems, programs, routines, objects and/orelements, remote control systems, or mixtures and combinations thereof.In other embodiments, if the timed hold is brief, then the processingunit causes an attribute to be adjusted to a preset level. In otherembodiments, if the timed hold is continued, then the processing unitcauses an attribute to undergo a high value/low value cycle that endswhen the hold is removed. In other embodiments, the timed hold causes anattribute value to change so that (1) if the attribute is at its maximumvalue, the timed hold causes the attribute value to decrease at apredetermined rate, until the timed hold is removed, (2) if theattribute value is at its minimum value, then the timed hold causes theattribute value to increase at a predetermined rate, until the timedhold is removed, (3) if the attribute value is not the maximum orminimum value, then the timed hold causes randomly selects the rate anddirection of attribute value change or changes the attribute to allowmaximum control, or (4) the timed hold causes a continuous change in theattribute value in a direction of the initial motion until the timedhold is removed. In other embodiments, the methods further comprise:sensing second motion including second motion or movement propertieswithin the active sensing zone of the motion sensors, producing a secondoutput signal or a plurality of second output signals corresponding tothe second sensed motion, converting the second output signal or signalsvia the processing units in communication with the motion sensors into asecond command function or a plurality of second command functions, andconfirming the selection based on the second output signals, orprocessing the second command function or the second command functionsand moving selectable objects aligned with the second motion directiontoward the selection object or become differentiated from non-alignedselectable objects and motion continues until a second target selectableobject or a plurality of second target selectable objects arediscriminated from non-target second selectable objects resulting inactivation of the second target object or objects, where the motion ormovement properties include a touch, a lift off, a direction, an angle,a distance/displacement, a duration, a velocity, an acceleration, achange in direction, a change in angle, a change indistance/displacement, a change in duration, a change in velocity, achange in acceleration, a rate of change of direction, a rate of changeof angle, a rate of change of distance/displacement, a rate of change ofduration, a rate of change of velocity, a rate of change ofacceleration, stops, holds, timed holds, and/or mixtures andcombinations thereof. In certain embodiments, sensing motions includingmotion or movement properties of two or more animals, humans, robots,and/or parts thereof within the active zones of the motion sensors,producing output signals corresponding to the motions, converting theoutput signals into command function or confirmation commands orcombinations thereof, where the start functions activate a plurality ofselection or cursor objects and a plurality of selectable objects orattributes upon first sensing motion by the motion sensor and selectableobjects aligned with the motion directions move toward the selectionobjects or become differentiated from non-aligned selectable objects andthe motions continue until target selectable objects or pluralities oftarget selectable objects are discriminated from non-target selectableobjects resulting in activation of the target objects and theconfirmation commands confirm the selections.

Systems Including Controller Apparatuses

The inventor has found that controller apparatuses may be fabricatedthat detect motion and determine motion or movement properties tocontrol physical or real objects, physical or real objects navigatingthrough real world environments, virtual or augmented reality objectsrepresenting real objects in virtual or augmented representations ofreal environments, virtual or augmented reality objects in virtual oraugmented reality environments, and/or virtual or augmented realityenvironments or attributes associated with any of these environments.The inventor has found that the apparatus may be in the form ofapparatuses including a plurality of sensors, a sensor array and/or aplurality of sensor arrays, communication hardware and software, and atleast one processing unit (generally, a digital processing unit) incommunication with the sensors or sensor arrays and the communicationhardware, where the sensors or arrays are capable of detecting motionand determining motion or movement properties in 1 dimension (e.g., x,y, z, t, θ, φ, etc.), 2 dimensions (e.g., xy, xz, yz, xt, yt, zt, rt,rθ, rφ, θt, φt, etc.), 3 dimensions (e.g., xyz, rθh, rθφ, etc.), 4dimensions (e.g., xyzt, rθht, rθφt, etc.), or higher dimensions. Itshould be recognized that in virtual or augmented reality environments,the dimensionality may be higher than 4, while in real environments, thetime-space has only 4 dimensions, while the objects may have may moredimensions associated therewith, where the dimension may be attributesor parameters defining the object. The controller apparatuses of thisdisclosure may be used to control real devices such as manned orunmanned planes, drones, robots, boats, motor vehicles, trains,submarines, matter, space (and any attributes associated with these) andany other device that is capable of moving on land, sea, sky, outerspace, or mixtures and combinations thereof. The controller apparatusesmay also be used to control virtual or augmented reality objectsrepresenting real devices or attributes or control virtual or augmentedreality objects that exist on in virtual or augmented realityenvironments.

Embodiments of the systems of this disclosure including apparatuses inthe form of 3D constructs (solid, hollow, or mixture thereof) includingat least one processing unit (e.g., a digital or analog processingunit), one or a plurality of sensors or sensor arrays, and communicationsoftware and hardware. The 3D constructs are designed to be held by auser. In certain embodiments, the sensors and/or sensor arrays includeat least one gyroscope and at least one accelerometer. In otherembodiments, the sensors or arrays may also include pressure sensors,temperature sensors, humidity sensors, field sensors, magnetometers,compass(es), optical sensors (UV, visible, near infrared (NIR), infrared(IR), microwave, radio frequency (Rf), etc. sensors), acoustic sensors,any other sensor, or mixtures and combinations thereof. In otherembodiments, the 3D constructs including regular 3D constructs such asspheres, ellipsoids, cylinders, prisms, pyramids, cubes, rectangularsolids, icosahedrons, dodecahedrons, octahedrons, cones, tetrahedrons,or any other regular 3D construct, or irregular 3D constructs such asdistorted and/or irregular versions of the regular 3D constructs.

Embodiments of the sensors and/or sensor arrays are configured in or onthe solid object so that they are capable of sensing motion and motionor movement properties, when the 3D object is moved. The motion ormovement properties including motion direction (linear, angular,rotational, etc., or mixtures and combinations thereof), motiondistance/displacement, motion duration, motion velocity (linear,angular, rotational, etc., or mixtures and combinations thereof), motionacceleration (linear, angular, rotational, etc., or mixtures andcombinations thereof), and/or changes in any of these properties overtime. In other embodiments, the apparatus is in the form of an objectincluding indentations or recesses for accommodating a user finger tips,fingers, or fingers and palm to facilitate holding of the apparatus. Inother embodiments, the systems of this disclosure may include two ormore such apparatuses being controlled by the same or multiple users.For example, a single user may be have one apparatus in each hand or twoor more users may have apparatuses in one or both hands so that thesystems of this disclosure detects motion from all apparatuses anddetermined motion or movement properties from all apparatuses andutilizes the collective motion to control physical or real objects,physical or real objects navigating through real world environments,virtual or augmented reality objects representing real objects invirtual or augmented representations of real environments, virtual oraugmented reality objects in virtual or augmented reality environments,and/or virtual or augmented reality environments. These may also workwith or include biometric, neurological, or other types of input orinfluencing forces.

Embodiments of the systems of this disclosure including apparatusesincluding at least one processing unit (e.g., a digital or analogprocessing unit), one or a plurality of sensors or sensor arrays, andcommunication software and hardware. In certain embodiments, the sensorsand/or sensor arrays include at gyroscopes, accelerometers, compasses,magnetometers, pressure sensors, temperature sensors, humidity sensors,field sensors, optical sensors (UV, visible, NIR, IR, microwave, Rf,etc. sensors), acoustic sensors, any other sensor, or mixtures andcombinations thereof. The sensors and/or arrays are configured to createa two-handed approach to navigate through virtual or augment realityenvironments or virtual or augmented reality representations of realenvironments, where the controllers are manifested in the virtual oraugmented reality environment as virtual control objects.

The present disclosure describes apparatuses that provide easier ways tocontrol real and/or virtual objects (e.g., real object include any realdevices such as drones, entertainment systems, motor vehicles, airplanes, etc. or virtual object include any virtual feature, construct,element, etc.). We have previously described the use of changes ofmotion and combinations of motion with touch, gestures and verbalinterfaces and modalities to select, scroll, activate, and controlobjects and/or object attributes. Sensors now available, such asaccelerometers, gyroscopes, compasses, GPS, near-field locators, opticalcameras and sensors, etc., allow us to provide new ways to interact withand/or control real objects, virtual objects, real and virtualenvironment content, and/or real or virtual environments.

Embodiments of the controller apparatuses of this disclosure comprises aphysical ball or sphere. This same controller may be used in or with avirtual environment or may be a virtual representation of a physicalcontroller to control virtual and/or real objects, attributes, zones,data, etc. The controller apparatuses may be in the form of a ball(virtual ball in a virtual environment) or a physical ball or any 3Dshape. The 3D shape may be symmetrical, asymmetrical, irregular, smooth,faceted, textured, colored, etc. In certain embodiments, the 3Dconstructs are symmetrical. In other embodiments, the 3D constructs arespherical. In other embodiments, the 3D constructs are generallyspherical having slight faceting with no sharp edges or corners. Thecontroller may include sensors providing for detecting location andchanges in location such as global positioning system (GPS) data, NFCdata, way point data, or any other location data and degrees of motionsuch as angular and/or rotational motion such as pitch, yaw, roll, etc.,linear motion up (+z), down (−z), left (−x), right (+x), in (+y), out(−y), any other motion, changes of any motion over time (velocity,acceleration, etc.), and/or any combination thereof. In certainembodiments, the controller apparatuses of this disclosure areconfigured to control a drone, unmanned vehicle, unmanned space craft,unmanned boat, unmanned air plane, unmanned submergible, unmanned airship, or other similar device, or for locomotion or influencingenvironments.

In certain embodiments, the ball controller may be activated by graspingit with the fingers (as opposed to holding it with an open palm) andmoving of the ball correlates to the movement of the drone. In a virtualenvironment, moving close enough or in proximity with a grasped palmposition, without having to actually be too close, would be theactivation). Once the ball controller is activated, moving it upwardsbegins the command to move the drone upwards. The distance and speedmoved upwards (or change in other movement properties) prescribes thevector(s), associated attributes, and any acceleration value. Beginningto move up begins the drone moving upwards, the further the ballcontroller is moved up, the faster the drone goes up. At the point theball movement is stopped (a hold function), or by relaxing the grip onthe ball controller, the current attribute and intensity continues. Achange in direction of the ball controller changes the direction of thedrone, based on real-time changes of vectorial motion of the ballcontroller, and intensity based on speed and distance of the ballcontroller moved. The range of motion correlates to the attributecontrol of the drone; i.e., once the ball controller is activated, −6 to0 to +6 inches (total of 12 inches) represents the full range ofattribute (such as 0 to 30 mph, or total distance ability of thedevice). It is preferable that the attribute ranges being in incrementsso small movements of the hand do not adversely affect the device.Compared with typical joystick controllers, where holding the sticksstill keeps attributes at a current value, holding the ball controllerstill keeps the attributes the same by relaxing the grip a thresholdamount.

Rotating the ball controller rotates the drone, based on acceleration,velocity, and direction of the ball controller motion. The systems ofthis disclosure may be designed so that rotation of the ball controller,while moving the ball controller cause the system to perform multipleselection and attribute control functions, synchronously, asynchronouslyor sequentially.

Another embodiment of the controllers of this disclosure may include aplurality of independently rotatable sections such as a top section(s),a horizontal middle section(s), a bottom section(s), a right section(s),a left section(s), a vertical middle section(s), other rotatablesections, and mixtures or combinations thereof. For examples, aspherical control apparatus may include a top section, a middle ringsection, and a bottom section, which may be rotated independently. Inother embodiments, the spherical controllers may include multiplesections and each section may include one or a plurality of rings.Controllers including multiple rotatable sections will provide morecontrol aspects. Twisting action may be used to leverage motion soinstead of moving the whole ball, a twist may cause the systems toexecute an identical or similar control function, without moving thecontroller, i.e., the controller stays in place. A twist may alsoindicate a different device or groups of devices to be controlled by thesame controller. The systems and methods may use twisting and moving tocontrol objects and/or object attributes.

In other embodiments, the controller may include a vertical orhorizontal member, such as a stick, rod, etc., attached, affixed orintegral with a top, side, or bottom of the controller. The constructsmay have a virtual extension of the physical extension pointing towardsthe ground or towards a desired location or direction for orientation orcontrols, such as a ray of light or a field distortion. The member maybe used to keep the controller at a specific distance from the ground orother surface so that all motion is relative to the specific location ofthe controller relative to the member. It may also be used to guide theuser in making decisions or providing other feedback or data forcontrols, decision-making, or locating of desired attributes or objects.Motion or movement about the member may also provide another layer ofmotion sensing and object and/or object attribute control.

The controller apparatuses may also be used in much the same way tonavigate through virtual or augmented reality environment and/or space,except instead of controlling a physical device moving through aphysical environment or a virtual or augmented reality representation ofa physical environment, the systems and methods use controller motion tomove through the virtual or augmented reality environment and/or spaceand/or to control VR/AR objects and VR/AR object attributes. Forexample, motion of the controller may cause a viewing angle to move(such as a camera through space), or may cause a scene to move inrespect to a viewers perspective. In this way by moving the controllerforward (away from the user), the environment may appear to move towardsthe user in the same perspective and leveraged way as descried above (12inches equals 0 to full speed of virtual “motion” of the scene). Bymoving the controller through an arc from left to right, the directionof turning of the environment is performed. By moving controller awayfrom the body at the same time, a forward moving and turning of theenvironment is performed. By moving the controller upwards, a moving ofthe sky or ceiling down is performed. All of these type of motions maybe done in combination, and in a small actual range of movement. Thesystems and methods may also response to the tilting of the controller.Such tilting may be combined with directional and rotational movement toprovide additional functionality. For example, moving, rotating andtilting may cause the system to move the physical object or VR/AR objectin the indicated direction and rotation at an angle or at an offsetdetermined by the tilt properties. A ring or other form of assistancemay be attached or part of the controller to assist in holding on to thecontroller.

The systems and methods of the disclosure may also include a previewfeature. The preview feature of the scene can be shown to represent themovement, while simultaneously, synchronously, asynchronously orsequentially showing the existing scene. With a hold, voice command,trigger or button push, a tighter grip or opening of the hand, the viewwould transition from the previous scene to the previewed scene in a“portal”, “jerk”, dissolve, or other transition display event so theuser is at the new desired location. For instance, if a wall had a doorand another room beyond the door, a grasping motion towards thecontroller and a movement of the controller towards the door (away fromthe body indicating moving forward), and a continued hold or evenfurther motion away from the body, would take the user into the nextroom with a “ghost” or wire frame look of the new location is overlaidon top of the existing color scene, so both scenes may be seensimultaneously, synchronously, asynchronously or sequentially, but withenough of a different look that the user can tell the difference betweenthe existing stationary scene and the moving, controllable previewedscene. Once the user “squeezes” the virtual device (or for a real devicepulls a trigger or holds the device upright), the scene immediatelytransitions from the previous screen to the new location.

This same control may be performed with no devices being represented(using just hand or body motions), with a virtual controller beingcontrolled by hand, body (eyes, etc.), or by motions of one or more realdevices. Using two hands, a hand and a real or virtual device, or tworeal or virtual devices, more controls may be provided. With two points,a plane or zone, or two or more planes or zones, or two sets of 3-axisplanes or zones, may be moved, controlled, and represented at once.Using the gaze of face, head or eyes could provide yet another set ofplanes and/or zones. Two hands may form two edges of a virtual plane. Aplane may be represented by one hand, possibly centered in a palm areaand rotates as the hand is rotates. Two hands may therefore representtwo different planes, and an intersection of these planes may be changedbased upon a relative distance between the two hands and/or a relativeangle formed by the two hands. Instead of moving two hands, two stickcontrollers, two ball controllers, or any other virtual or realcontroller. These may then be used to represent one previewed scene withone hand and another with the other hand, creating an entirely new wayto move from one location to another, or by combining previewsassociated with each hand, then instantly being “moved” to this newhybrid location with a selection event. One hand may represent a coloror an intensity or other attribute effect that provide overlaidinformation to the other hand displayed attributes. One hand laiddirectly over the other may perform a mirrored effect between the twowith a gradient of effects between the two. One hand may perform azoom-in or zoom out function while the other performs location selectionor movement. So one may preview where they want to go and scale theview. Of course this may be performed with one hand, but two may providea better experience.

Another benefit to this approach is that the apparent “horizon” orstable “line of site” remains for the viewer while a “ghosted”,foveated, or non-similar image is displayed simultaneously,synchronously, asynchronously or sequentially, so the user can virtuallymove through space without the nausea effects of moving the actualscene. This also allows the user to see where they have been (the actualscene) and where they are going (the preview), simultaneously,synchronously, asynchronously or sequentially. This same effect may beused to control a drone with an augmented reality set of glasses ordevice. On the glasses display, the image of the camera view of thedrone (or any other device) may be displayed, so that the user sees whatthe camera is seeing. By virtually previewing the area (say by using asatellite image to create a virtual “world” around the drone), one maybe able see through a virtual “eye”, while simultaneously,synchronously, asynchronously or sequentially seeing the real worldthrough a real camera “eye”. Once the previewed area is selected, thedevice may then move to a new location with the camera view lining upwith the previewed scene, or in whatever predetermined scaled amountdesired. This may be done for any attribute such as viewing angle,sound, amplitude, orientation, color speed, or any combination ofattributes. The same is true of head or eye tracking, or the ballexample above. These are two different embodiments of the sameprinciples.

Methods for Secondary Device for Object Control

In certain embodiments, the systems, apparatuses, and/or interfaces ofthis disclosure and methods implementing them include using one device,say a phone, to control a display of another device, such as a secondphone, where a menuing and controls of this disclosure installed on onedevice permits control the other device(s) and/or their associateddisplays, attributes, or hardware or software. This methodology wouldallow one object to control one or more objects even if the objects usedifferent operating systems, have different environments, and/or havedifferent hardware. This ability for one device or object to controlother devices or objects is another example of the use case of ourpredictive dynamic motion controllers.

Deliberate Movement Differentiated from Spontaneous or Non-DeliberateMovement

In certain embodiments, the systems, apparatuses, and/or interfaces ofthis disclosure and the methods implementing them include sensingdeliberate or intentional, generally predefined, movements, outputtingthe sensed movement as an output, and converting the output into acommand and control function including, without limitation, a selectfunction, an activate function, a scroll function, an attribute controlfunction, and/or combination thereof. The deliberate or intentionalmovements may be associated with eye tracking or head tracking motionsensors or with any other motion sensor or deliberate or intentionalmovements associated with a specific body part or member under thecontrol of an entity. The deliberate or intentional movements may be tomove an eye or the eyes across a displayed selectable object, then tochange a speed a predetermined amount so a desire function is invoked.For example, for the systems, apparatuses, and/or interfaces including adisplay and an eye-tracking sensor, when the user looks across aparticular object or a set of objects or stares at a particular objector set of objects, then a particular function may be invoked such as aselect function, a select and activate function or a select, activateand adjust attribute value function, but if the user looks across a faceof the object at a preset speed, then a particular function may beinvokes such as a select and activate function. It should be recognizedthat in the case of eye movement, the deliberate or intentionalmovements including its movement properties must be discernibly distinctfrom normal eye movement. In certain embodiments, the systems,apparatuses, and/or interfaces sense motion from one or more motionsensors and monitor the movement until the movement meets one or morecriterion sufficient to distinguish the movement from normal eyemovement threshold criteria are satisfied. For example, the deliberateor intentional movement may be a slow but continuous movement, a pauseat a corner and a look quickly towards another corner (diagonally), orsome other change of rate of speed or acceleration that isdistinguishable from normal eye movement.

Another example of deliberate or intentional movements may involvedifferentiating normal viewing behavior from viewing behavior that isdeliberate. Users typically do not look directly at a middle of adisplayed object, but rather look at the whole object or just below acenter, i.e., the users focus is not on the center of the object. Thus,a deliberate movement may be just to stare at a center of an object orto stare at some other location in an object; provided, however, thatthe movement is sufficient for the systems, apparatuses, and/orinterfaces to distinguish the movement from normal eye movement. Aperson may look at an object, and when it is determined by a sensor thatan object is generally being looked at, a center or centroid of theobject may be displayed differently (or just be active without appearingdifferently), such as a square or circle showing the centroid area sothat the systems, apparatuses, and/or interfaces may use the motionsensor output associated with looking into the area or volume or movingthrough this area or volume and converting the output into a command andcontrol function. Of course, the triggering area or volume may not bethe center, but may be another location within the object. Therefore,looking at or towards an object may cause the systems, apparatuses,and/or interfaces to pre-select the object, but only when the user movesthe gaze into the active area/volume (generally predefined) does thesystems, apparatuses, and/or interfaces invoke a particular command andcontrol function. Alternatively, the deliberate movements may involvemoving across a predefined area, where speed of the motion does notmatter, only that a traversal to a certain threshold is reached.Additionally, other movement properties (e.g., speed, velocity, and/oracceleration or changes of these) may be used as part of the predefinedmovement to invoke a particular function or functions. This sametechnique may be applied to users that have certain type of maladiesthat prevent them from smooth movement, the systems, apparatuses, and/orinterfaces may be tailored to determine difference(s) between normaluser movement and deliberate user movement even though the difference(s)may be subtle.

Constructs with Continuous Properties

In certain embodiments, the systems, apparatuses, and/or interfaces ofthis disclosure and the methods implementing them may utilize constructshaving continuous properties (e.g., continuous values analog instead ofdiscrete values digital). In such environments (all objects arewaveforms that are capable of interacting), the movement may navigatethrough the continuous properties with a change in movement or adeliberate movement may result in the selection of a particular value ofa continuous property or a set of continuous properties. Thus, waveformand waveform interactions may be manipulated, adjusted, altered, etc.and viewed. Additionally, given interaction patterns may cause thesystems, apparatuses, and/or interfaces to invoke a particular functionor set of functions. Attribute may be a subset or other attribute of anobject, but may also be associated with a change in a waveform, that isdifferent from scrolling, in that scrolling must have integer values (orstops along a path). It like a guitar, where scrolling would be movingthrough frets, but sliding the string sideways (bending) the stringproduces frequency changes with no preset integer values, where systems,apparatuses, and/or interfaces may use both outputs to invoke adifferent function or set of functions, which may be predefined ordetermined from context on the fly.

Real-Time Prediction of User Intent

People move typically in a straighter line, and faster when they knowwhat they want or are choosing something. In certain embodiments, thesystems, apparatuses, and/or interfaces may be used to predict to acertain probability, what a particular user choice may be based on howfast and/or straight the user moves towards a particular selectableobject. In some cases, such as movement of the thumb, where the movementcomprises rotating about a thumb joint, the motion may be arcuate, andmoving in a non-arcuate manner may be seen as more intentional, thusproviding a higher probability. Other things that may affect theconfidence of making a selection (or the probability), include proximity(closer to one object than another), time spent in different zones or inproximity to objects, changes in directions, slowing down as approachinga particular object (changes in direction, distance, duration, speed,velocity, acceleration, etc.) such as decelerating when moving towards aparticular letter on a keyboard, then moving away at an increasedacceleration (after choosing a letter on a keyboard and moving to thenext) and rates of these changes.

The systems, apparatuses and/or interfaces may improve real-timeconfidence determinations by using artificial intelligence (AI) routines(including algorithms, parameters and elements). The AI routines mayimprove confidence determinations based on confidence data including alltrainee performance data, trainee specific type performance data,historical performance data, environmental data that affects trainee ortrainer performance, or contextual data that affects trainee or trainerperformance. These data are stored in libraries and/or databases may beused to update or modify the routine, routine components, task, and/ortask components or may be coupled with the above movement properties toenhance predictive confidence determinations and to update or modify theroutine, routine components, task, and/or task components. The AI mayalso be used to forensically, or in real time, identify parameters,events or data. The AI routines may be based on any AI methodologyincluding, without limitation, neural networks AI routines,gradient-based learning AI routines, deep Boltzman machine learning AIroutines, large-scale unsupervised learning AI routines, deepconvolutional activation feature for generic visual recognition AIroutines, convolution neural network AI routines, recurrent neuralnetwork AI routines, back-propagation neural network AI routines,stochastic gradient descent AI routines, learning rate annealing oradaptive learning rates AI routines, dropout AI routines, maximumpooling AI routines, batch normalization AI routines, long short-termmemory AI routines, skip-gram AI routines, continuous bag of words AIroutines, transfer learning AI routines, fuzzy logic AI routines, expertAI routines, perceptron AI routines, decision management AI routines,Naive Bayes AI routines, support vector machine AI routines, linearregression AI routines, logistic regression AI routines,K-nearest-neighbor AI routines, K-means AI routines, decision tree AIroutines, random forest AI routines, CART AI routines, a priori machinelearning algorithm based AI routines, principal component analysis AIroutines, CatBoost AI routines, Iterative Dichotomiser 3 AI routines,hierarchical clustering AI routines, back-propagation AI routines,AdaBoost AI routines, deep learning AI routines, gradient boostingalgorithm AI routines, Hopfield network AI routines, C4.5 AI routines,any other AI routine, any waveform analysis and use routine, or anycombination of AI routines.

Self-Centering User Interface (SCUI)

In certain embodiments, the systems, apparatuses, and/or interfaces ofthis disclosure and the methods implementing them relate to novelself-centering interface (SCUI) for controlling objects (software,hardware, attributes, waveforms or any other selectable, scrollable,activatable, scrollable or otherwise controllable thing) such ascontrolling drones through head motions using head motion sensors. Forexample, picture a compass rose with a hole in its middle and dividedinto 4 quarters: NE, NW, SW and SE. As the user moves leftward in the SWquadrant, the systems, apparatuses, and/or interfaces may cause thedrone to move to the left and a distance of the movement to the leftcontrols the speed of the drone's movement to the left. Thus, thefurther the user moves to the left within the SW quadrant the faster thedrone moves to the left. Similarly, as the user moves rightward in theSE quadrant, the systems, apparatuses, and/or interfaces may cause thedrone to move to the right and a distance of the movement to the rightcontrols the speed of the drone's movement to the right. Thus, thefurther the user moves to the right within the SE quadrant the fasterthe drone moves to the right. In this way, the user may use a pair ofglasses (such as AR/VR/MR glasses, etc.) and see the drone, and move thedrone while using a semi-transparent UI design, when using anintentional speed of head movement, i.e., deliberate head movement. So,by moving quickly, the UI may not cause the drone to move as thesystems, apparatuses, and/or interfaces may determine that such movementdoes not represent a deliberate movement sufficient for drone control.Thus, if the movement is determined by the systems, apparatuses, and/orinterfaces to be a deliberate movement, then the UI may cause the droneto undergo are corresponding movement. By moving in a specificdeliberate manner, a menu may be activated, the view centered along thefocus or gaze direction (self-centering) and the menu objects orelements arranged in a spaced apart configuration (e.g., concentrically)about a center of the user head or eye position, i.e., arranged aboutthe gaze point. In this donut compass rose example, when the gaze is inthe center, or donut hole area, the systems, apparatuses, and/orinterfaces cause the drone to transition into a stationary state, whichmay be a hover state or a state of constant motion-based on the last setof head/eye movements. The systems, apparatuses, and/or interfaces maydiscriminate between a hover state and a constant motion state base onthe duration of the gaze (duration of a timed hold), on where in thecenter area the gaze is fixed. Moving left and right (x-axis) moves thedrone left and right. Moving up and down (y axis) move the drone up anddown. Moving in a combination of x and y movement, moves the dronesimilarly. Additionally, other movement within different quadrants suchas the movement within NW or NE quadrants may control rotation of thedrone on its axis, left or right, respectively, or may control pitch,yaw, roll, or other motions.

In other embodiments, the systems, apparatuses, and/or interfaces ofthis disclosure and the methods implementing them relate to novel userinterfaces comprising three different control object formats: screenlocked, world locked absolute, and world lock relative. Screen lockedmeans that an object, a plurality of objects, an attribute, and/or aplurality of attributes remain in the user field of view at all timesregardless of where in the “world” the user view is. World lockedabsolute means that an object, a plurality of objects, an attribute,and/or a plurality of attributes may become associated with ortransitioned to a specific world view object or a specific world viewlocation remain fixed to that object or location and do not move.Therefore, if the user movement moves that view so that the object orlocation moves outside of the current view, then the control objectsand/or attributes associated with the object or location will be nolonger visible. World locked relative means that an object, a pluralityof objects, an attribute, and/or a plurality of attributes may beassociated with or transitioned to the world view, but the object, theobjects, the attribute, and/or the attributes may follow the user gaze,but lag behind so that they may not be accessible until the movementstops or stops for a specific period of time. For drone controls,certain drone controls may be screen located, while other drone controlsmay be world locked absolute, while other may be world locked relative.For example, a target and/or target attributes may be world lockedabsolute, a drone position controls for moving the drone along a path tothe target may be world locked relative, and camera controls or weaponcontrols may be screen locked. Of course, the user may change theobjects and attributes that are screen locked, world locked absolute, orworld locked relative.

Putting these concepts together, sensing a deliberate movement causesthe systems, apparatuses, and/or interfaces to activate the UI or tobegin user interaction with the UI and causes an image of the drone toappear in the world view. The UI comprises the three locked formats.Then, sensing movement to the left within the SW quadrant, the systems,apparatuses, and/or interfaces causes the drone to move left, where thespeed of drone movement to the left is controlled by the distance thesensed movement of the user to the left within the SW quadrant. As thedrone moves, the screen locked object, objects, attribute, and/orattributes move with the users; the world locked absolute object,objects, attribute, and/or attributes remain fixed to an object in theworld or a location in the world; and the world locked relative object,objects, attribute, and/or attributes track the movement of the drone.The tracking may be appear as it the object, objects, attribute, and/orattributes are screen locked they move in direct correlation to thedrone, or they are move at a slower rate or they move so that only afteruser movement stop that they move back into the user view. Optionally,the world locked relative object, objects, attribute, and/or attributesmay move in front of the drone so that the has a preview of the dronescourse and may adjust it accordingly. When the user head or eye movementstops at a gaze point (user gaze at a fixed location in the world view),then the drone movement will either stop or the drone continues to movein accord with the movement at the time the user movement stops, where atype of gaze duration, gaze center, etc., determines whether the gazecause the drone to hover in place or continue to move in accord with thelast movement properties. In certain embodiment, once a fixed gaze isdetected, then the world locked relative object, objects, attribute,and/or attributes, which have been following the user movement, catchesup to the gaze point, and become centered about the gaze point. Becausethe UI is controlling the drone, the drone now centers itself inalignment with the UI, which is centered around the gaze point. In oneembodiments, the UI lags slightly behind the gaze point, and the dronelagging slightly behind the UI.

This same UI may also be used to control z-axis motions by either using3D sensor data (from head motion sensor or other motion sensors), or byusing a unique 2D construct that provides 3D controls. An example ofthis is the same compass rose (or circular/radial UI menu/controller)with a donut hole, but now adding a designated z-axis area as describedherein. In one embodiment, the UI is in the shape of a funnel as setforth herein, providing a slim, pure z-axis control wedge zone centeredwithin the z-control wedge. Moving towards or away from the center ofthis z-zone moves the drone along the z-axis. The UI is divided into twoparts, with a dead zone. The center area provides 3D x/y/z axiscontrols, while the outer part of the funnel is 2D and provides only x/ycontrol (as described above). In the 3D area, if the user moves out ofthe Z-zone, but remains in the inner section, then the motion representsa combination of x, y and z. If the user moves into the outer zone, onlyx/y controls are provided.

Systems, Methods and Interfaces Including at Least Two User FeedbackDevices

More particularly, embodiments of this disclosure relate to systems,apparatuses, and/or interfaces and methods for implementing them on orin a computer, where the systems, apparatuses, and/or interfaces andmethods of interacting with 3D or n-dimensional (nD) environments usingat least two user interface devices, where the systems, apparatuses,and/or interfaces include at least one motion sensor, at least oneprocessing unit, and at least two user feedback unit for controlling,interacting, and manipulating motion in 2D, 3D, and/or nD environments.

Analytics Using the Same Motions that Control Things

In certain embodiments, the systems, apparatuses, and/or interfaces mayuse historical data to predict user intent and cause actions (such asselections) to happen faster without have to move to an object. Thus, byanalyzing past user behavior and movement characteristics, the systems,apparatuses, and/or interfaces may be able to more quickly, which objectaligned with a particular movement is more likely the target. The samevectors that change with speed and direction (and these changes providecontrols) also tell us many things about the user. For instance,scrolling back and forth (say x-axis movement) between two out of fiveitems, then moving towards a particular object (say y axis movement),selects and activates that object. But knowing the other objects letsthe systems, apparatuses, and/or interfaces classify alternate choices,that may be ranked based on historical data. The use of analytics mayfind particular application in advertising or training methods using themotion-based systems, apparatuses, and/or interfaces of this disclosure.

In other embodiments, systems, apparatuses, and/or interfaces may usethese predictive methodologies that cause objects to move towards theuser or a selection object to predict zones for foveated rendering. InVR or other mixed reality environments, graphics rendering is extremelytime consuming. To compensate for this, graphics rendering at thehighest resolution is generally restricted to an area or areasassociated with a center of vision. By restricting the high resolutionrendering to these areas provides the user with a good experience. Thus,the high resolution graphics rendering doesn't need to be performed onzones, areas, or volumes not being looked at. In this way, prediction ofwhere the user will be looking may assist in foveated rendering, so thatpart of the display may be rendering the predicted zones, so the usersees no apparent delay in rendering.

In other embodiments, the systems, apparatuses, and/or interfaces ofthis disclosure and the methods implementing them, where the systems,apparatuses, and/or interfaces include at least one eye and/or headtracking sensor, at least one processing unit, and at least one userfeedback unit. The systems, apparatuses, and/or interfaces permit twodifferent pinning modes. The first pinning mode is that the trackingsensor includes information about objects displayed in a tracking basedmanner viewable at a left and right edge of the viewing plane. Theseobject may be selected by moving the head and/or eyes toward thetracking pinned objects causing them to appear in the center of thefield so that they can be controlled by further head and/or eyemovement. As the user views a real world object in a real worldenvironment, an object in a AR environment or an object in a VRenvironment, the user may transition the selection format from atracking pinned format to a world pinned format. In the tracking pinnedformat, the selection and control function for the object under thecontrol of the systems, apparatuses, and/or interfaces remain with thetracking sensor and may be accessed at any time, but once the user seesan object pauses at the object or moves in a predetermined manner towardthat object, the systems, apparatuses, and/or interfaces pins the objectcontrol functions to the object. The pinning may be permanent orrelative. Permanent pinning ties the control functions to the object sothat you may return to the object to be able to control its attributes.Relative pinning means that the object control function travel with theworld view either directly or with a lag as it follows the eye and/orhead movement.

Systems, Apparatuses, and/or Interfaces and Methods Using Blob Data

The inventor has found that movement based systems, apparatuses, and/orinterfaces and methods implement them, where the systems, apparatuses,and/or interfaces include at least one sensor, at least one processingunit, at least one user cognizable feedback unit, and one real and onereal or virtual object or a plurality of real and/or virtual objectscontrollable by the at least one processing unit, where the at least onesensor senses blob data associated with touch and/or movement on orwithin an active zone of the at least one sensor and generates an outputand/or a plurality of outputs representing the blob data, and where theat least one processing unit converts that blob data outputs into afunction or plurality of functions for controlling the real and/orvirtual object and/or objects.

In certain embodiments, the systems, apparatuses, and/or interfaces ofthis disclosure and methods implementing them include using blob data asa source of movement data for analyzing, determining, and predictingmovement and movement properties, where movement is understood to meansensing movement meeting a threshold measure of motion by a motionsensor, a plurality of motion sensors or an array of motion sensor foruse in motion-based object control, manipulation, activation and/oradjustment. Blob data comprises raw motion sensor data representingsensor elements that have been activated by presence and/or movementwithin an active area, volume or zone of the proximity and/or motionsensor(s). In the case of a touch screen including a large plurality oftouch elements, touching the screen produces raw output datacorresponding to all touch elements activated by the area of contactwith the screen and comprise the blob data for touch screen or otherpressure sensors or field density sensor or sensor including activatablepixels or any other sensor that include elements that are activated whena threshold value associated with the element is exceeded (pressure,intensity, color, field strength, weight, etc.). The term activate as itrelates to touch elements means that touch elements within the contactarea produce touch element outputs above a threshold level set either bythe manufacturer or set by the user. For other types of sensors,movement within an active sensing zone of the sensors (e.g., areas for2D devices, volumes for 3D devices) will activate an area and/or avolume within the zone. These areas and volumes represent the “blob”data for each type of device and comprises elements having a valueexceeding some threshold value for activating the elements. For imagebased sensors, the activate elements will generally comprise pixelshaving a threshold value of pixel values. For capacitive sensors orinductive sensors or electromagnetic field (EMF) sensors, the blob datawill relate to areas or volumes corresponding to sensor elements thatmeet a threshold output for the sensors.

The blob data (activate element area or volume) will change with changesin contact, pressure, and/or movement of any kind. The blob datarepresents an additional type of data to control, manipulate, analyze,determine, and predict movement and movement properties. The blob datamay be used to identify a particular finger, to differentiate betweendifferent fingers, to determine finger orientations, to determinedifferences in pressure distributions, to determine tilt orientations,and/or to determine any other type of change in the blob data.

In the biokinetic applications, the blob data with or without theaddition of filtered data (center of contact, center of pressure, orother types of centroid data) may be used to create a proportionateand/or unique user identifier. Not only may blob and centroid data bebiometric identifiers, but the relationship between the two is a moreunique biometric, or electro-biometric identifier. The systems,apparatuses, and/or interfaces of this disclosure may also includesensing, determining, and analyzing the blob data and determining andanalyzing filtered data or centroid data for use in analyzing,determining, and predicting movement and movement properties for use inmotion-based object control, manipulation, activation and/or adjustmentof this disclosure. For example, a user places a thumb on a phone touchscreen. In doing so, the blob data may be used to identify which thumbis being used or to confirm that the thumb belongs to a particular user.If the touch screen also may include temperature sensors, then the blobdata may not only be used to differentiate and identify particularthumbs (or fingers, irises, retinas, palms, etc.) alone or inconjunction with other movement data based on a shape of the blob dataor output signal and a direction to which the blob data or blob data andcentroid data may be pointing or oriented. This technique may be used todirectly turn a knob using a pivoting movement versus using movement ofa centroid, where the thumb is represented as a point and movement ofthe centroid from one point to another is used to determine direction.Using blob data allows the user to select zones, control attributes,and/or select, scroll, activate, and/or any combination of these, thesystems and methods of this disclosure simply by pivoting the thumb.Then moving the thumb in a direction may be used to activate differentcommands, where the blob data movements may be used to accentuate, toconfirm, to enhance, and/or to leverage centroid data. For examples,pivoting the thumb while in contact with the touch screen results inblob data that may be used to determine finger orientation and/or tilt,allowing the user to select between groups or fields of objects (forexample), or through pages of data or objects. Once the user scrolls andselects a particular group or field, further movement results in adifferent set of set of controls, instructions, commands, attributes,etc. The systems and methods may use the blob data to “see” oranticipate movement attributes (direction, pressure distribution,temperature distribution, speed (linear and angular), velocity (linearand angular), acceleration (linear and angular), etc. The systems andmethods may use the blob data, the centroid data or a combination of thetwo types of data to analyze, determine and/or predict or anticipateuser movement. The transition from blob data to centroid data may alsobe used to see or anticipate user intent. For example, as a user twistsor pivots the thumb, then begins to move towards an object, zone orlocation, the thumb may begin to roll in a lifting motion, rolling uptowards the tip of the thumb, providing less of a pattern and more of atypical centroid touch pattern on the screen. This transition may alsoprovide user intent through not only movement in an x/y plane, but alsoproviding shape distinctions that may be used for commands and otherfunctions. The rocking of the thumb or finger (rocking from a flatorientation to a tip orientation) may also provide z-axis attributes orfunctions. This may also be combined with movement while rocking. In 3Denvironments, the blob and/or centroid data (along with other movementattributes such as direction, pressure distribution, temperaturesdistribution, etc.) may be used, but instead of blob data, pixilation in3D in any environment, or volumetric differences (sensed in any way)along axes (plural) may be used in the same way as blob and/or centroiddata to analyze, determine, anticipate, and/or predict user intent.These aspects may also be seen or used as a “field” of influencedeterminative. In these embodiments, temperature may be used for anumber of different purposes. First, the temperature data may be used toensure that the motion sensor is detecting a living person. Second, thetemperature data may be used as data to insure that the user sensedwithin the active zones of the sensor or sensors is indeed the user thathas access to the systems and methods on the particular device. Ofcourse, temperature data is not the only data that the sensors maydetermine. The sensors may also capture other user specific data.

In certain embodiments, the systems and methods of this disclosureinclude controlling a hologram remotely or by interacting with it.Pivoting the hand in parallel with a field may provide one control,while changing an angle of the hand may be perceived as a “blob” datachange, a transition to centroid data, or a combination thereof. Thistransition may also be represented on a display as going from a blob toa point, and the transition may be shown as a line or vector with orwithout gradient attributes. Putting these into the hologram example,changing from blob data to centroid data, and seeing a vector and agradient of change of volume or area along the vector may be used tochange the display in the hologram of a shoe (for example) so the shoemay change size and direction according to the movement of the user.This methodology may be performed in any conceivable predetermined ordynamically controllable way, where attributes may be any single orcombination of intent, attribute, selection, object, command or design.These movements and/or movement attributes may be simultaneously orsequentially used in any environment, and in whole or part, and includegradients of attributes based on changes of perceived mass, pressures,temperature, volume, area, and/or influence. These changes may be sensedand defined by any sensor or software reproduction ability (software maybe used to replicate movement or the effects of movement). This alsoallows for a 2D sensor to provide 3D controls. All this may also be usedto determine unique BioKinetic identifiers as well and in combinationwith these attributes.

In certain embodiments, the systems and methods of this disclosureinclude using blob data to orient a menu appropriately, where the blobdata comprises raw sensor output data based on a number of sensingelements being activated above the threshold activation. For example, inthe case of a touch screen, when a user touches the screen with a fingertip or other part of a finger, the sensor generates a blob of datacomprising all sensing elements activated (based on some thresholdactivation value). The data is generally used to determine a centroid ofthe contact and that value is then used in further processing. However,the blob data may be used not only to differentiate different users, butmay also be used to predict or anticipate user movement and ascertainmovement and changes in movement. By knowing which thumb or finger islocated at what area of the screen, the displayed menu upon a touch orentry into a sensor area may be positioned to provide a best heuristicsor positioning based on the touch area and or user movement. Forinstance, touching the right thumb on a right side of a phone screen ina lower quadrant may signal the systems or methods to display a menualong a radius just above the thumb, while an angle of the thumb whentouching a middle of the screen may result in displaying a radial menujust below the thumb if the thumb was pointing upwards towards anopposite corner, or above the thumb if the thumb was pointing towards abottom left corner.

In certain embodiments, the systems and methods of this disclosureinclude one menu appearing when touching an upper part of the screen anda different menu appearing when touching a different part of the screensuch as a lower part of the screen. If the finger is flat and not angledwhen touching the screen, different menus may be activated. So theposition of the finger, finger angle, finger direction, finger pressuresdistribution, and/or combinations thereof may result in different menusets, object sets, attribute sets, command sets, etc., and/or mixturesof combinations thereof for further processing based on movement data.Of course, all of these concepts may be equally applied to 2D, 3D, 4D,or other multi-dimensional environments both real, augments and/orvirtual.

Systems and Methods Using Bread Crumb Procedures

In certain embodiments, the systems and methods of this disclosureinclude using “bread crumbs” or “habits” to determine direction ofmovement in an active zone or field of a sensor, of a plurality ofsensors, and/or of a sensor array. When a user moves towards a desiredlocation on a screen of a phone, especially across the screen to make atouch event, the sensor(s) will begin to “see” data associated with theuser's movement, but not necessarily in a continuous manner. Instead,the sensor(s) will see a series of points, with increasing frequency,intensity, and/or coverage area, and will begin to be sensed as the usermovement comes closer to “contact” with a desired screen location. Thisdata may be used to determine speed and direction, which in turn may beused to predict or anticipate user intent, which objects or attributesare active for choosing attributes rather than objects first is anotherapplication that you have filed. This provides a verification aspect sothe objects and/or attributes may be selected before a physicalconfirmation occurs (a touch event), or to cause objects and/orattributes to begin to respond (with color changes, sounds, tactilefeedback, shape, animations, etc.) before a confirmatory touch or actionoccurs. In this way, movement and then a touch may represent a uniquesignature or identifier as well. It should be recognized that the breadcrumbs or habits may be positive attributes and/or reactions or negativeattributes and/or reactions.

In certain embodiments, the systems, apparatuses, interfaces, and/ormethods of this disclosure include a user performing a movement orgesture then verbally identifying or confirming what attribute, command,or function to associate with the movement or gesture. This may besimultaneously or sequentially performed. Again, in the context of thisdisclosure, simultaneous means events that occur concurrently or eventthat occur in rapid succession within in a “short” time frame (e.g., ashort time frame is between about 1 as and about 1 s), whilesequentially means that the actions occur sequentially over a “long”time frame (e.g., a long time frame is between about 1 s and about 10s). For example, a user moving in an upward direction, while saying“volume up” results in controlling and increasing a volume of a sound. Auser may instead say “base” or “base up”, and a base intensity increasesinstead of the volume.

In certain embodiments, the above describe aspect may be used as asecurity identifier, where a movement and a voice command may be used tounlock a locked menu, object, and/or attribute or act as a uniqueidentifier for activating a menu, object, and/or attribute. By movingwith a right finger or thumb from left to right, and saying “open”, alocked phone may be unlocked, or any other command or function mayoccur. Both aspects may provide for biometric identification as well.Another example is facial recognition plus a specific gesture or dynamicmotion event. These changes may be sequential changes collected over along time frame and/or simultaneous changes collected over a short timeframe allowing further refinement of user identification, verificationand/or authentication. This may also include multiple touches or sensedpoints, multiple words or commands, or any combination of these. Insteadof words, sounds, notes, or any audible or other kind of wave form maybe used. Touching a zone or location on a screen, while saying a desiredattribute, command, or any other desired choice is another way this maybe used. Another benefit of this is the ability to quickly associatecommands or attributes (scrolls, selections, actuations, or attributes),training a system or interface in an easy way. In this way, controls caneasily be customized to the user preferences.

Another example of this methodology is to use an area of a touch on thescreen. By touching the upper right quadrant of the screen (or moving inthat direction) and saying “travel”, the system may be trained orprogrammed so that this touch may display a travel menu of objects orother attributes. By touching or moving in (or towards) the bottom rightquadrant and saying “food”, a menu of restaurants may be displayed. Fromthat point on, touching or moving towards the associated location orarea may provide a different menu, selection or attribute than movingtowards or touching a different area. In another example, saying“lights” may invoke a display of controllable lights on a mobile deviceor in a virtual or augmented environment. By aligning a finger or gazewith the “lights” object, the user may say “hallway” and the hallwaylight would move under the user's finger or gaze, just as if it had beenselected by moving towards it (as described in other applications). Acolor wheel may display and the user move into the color wheel to selecta color, or the user may say “warm” and a sliding scale of yellow-basedcolors would move or appear under the user's finger, which the usercould then move to select the desired hue. By having a motion-basedarchitecture already established, any motion may be replaced by a voicecommand, and any voice command may be replaced by a motion, or acombination of the two. As the User moves towards the bedroom light iconand says “warm”, the sliding scale would show under the user's finger(or gaze) and immediate attribute control would be possible. This isalso true in 3D environments such as an augmented or virtual realityenvironment, where gestures or movement may be associated with controls,selections, menu items or attributes by performing the desired gestureor motion and saying (simultaneously or sequentially) what theassociated attribute and/or selection is. This may be performed invirtual space or on real displays, so that areas or volumes or locationsin 2D and 3D space may operate in the same way.

In certain embodiments, the, apparatuses, interfaces, and/or methods ofthis disclosure include locating an object at a point where it may havebeen before, or a 3D camera in a structure so it is the optimal distancefrom walls or other objects in a space. One way of doing this is to takea phone (or any device with sensors) and touch a wall or come closeenough to be considered a threshold event (for example) with the phoneand a trigger of some kind (touching a control object on the phone orsaying “start” or other kind of triggering command, and begin to walktowards a perceived location in the middle of a room. The phone displaysa visual “chord” or vector from where the wall was touched to yourlocation. This may be done by using the compass, magnetometer,gyroscope, accelerometer, or any other sensor of the phone and the stepsas measure by other sensors of the phone (such as changes in theaccelerometer data of the phone). Repeating this with each wall, or atleast one or two points, and as the user moves, the intersection ofthese vectors can be determined and seen on a screen. By running spatialalgorithms, the central part of the room can be determined. This canthen be repeated later using different wall points to locate the centerat a later point. By also using the distance from each wall or usingcorners or a wall at a specific height, accuracy is greatly enhanced.This ability to “drag” a set of vectors makes it easy for a user to moveand locate the point they wish to recreate or find by using a display,processor and sensor combination. A central point or center of area canbe determined as well as a previous point. In this way, a camera may beset at the same point over time to get sequential pictures from the samelocation.

The zoom level may also be set by a hold or other event, such as atouch, a “bounce” or “jerk” movement or gesture (of any kind), or verbalcommand, or any combination of these in motion-based processing systemsand methods described herein.

Zoom levels may be set by common parameters that are sensed, i.e., adistance from the face (most people will have a preferred distance toread, and this may be preset, or set at some point), and then holdingthe device at that point (within a range) triggers the capability.

Certain real content (real-world objects or images seen through a cameraor sensed by other sensors in real time, sounds heard in real time by amicrophone or other sensor), imaged content (web pages, images, videos,any other content shown on a device, or any kind of content not beingseen as real content), virtual content (animated or extra image contentthat is associated with imaged or real content i.e., when looking at apicture of a shirt either through the camera of a device, or whilelooking at an image of a shirt from a web site may trigger an animated3D person wearing the shirt and walking on a sidewalk to provide furthercontent to a user), or any data (commonly called metadata) associatedwith any content any of these, or combination of these, may trigger aset point or to cause the systems to activate with a certain preset zoomand/or pan level.

Multi-User Combinational Controls

Embodiments of this disclosure also relate to systems and methods thatinclude motion-based sensor subsystems that support a plurality of usersinteracting with the sensor subsystem so that the systems are methodschange the attributes of the displayed object(s), attribute(s), and/orenvironment(s) by combining the user interactions resulting in acombinationally changed object(s), attribute(s), and/or environment(s).For example, if the display is a hologram or picture, the sensorsubsystems will sense movement by each user as each user interacts witha motion-based processing subsystem of this disclosure. The systems andmethods of this disclosure then combines the user interactions to changethe picture or hologram to reflect a combination of the individualsystem interactions. For example, if the multiple users are interactingwith a hologram and each select a color for the hologram, then the coloractually displayed will constitute a color that is an addition of thecolors selected by the users. If one user selected blue and anotherselects red, then the resulting color will be purple. The actual colormay also be based on a “strength” of the selection or a “manner” ofselection or a “timing of the selection. Thus, if the blue user is firstto select, then the resulting color may be a bluer purple, while if thered user is first to select, then the resulting color may be a redderpurple. If the blue user, make a more the blue selection immediately,while the red user took longer to select, then the resulting color maybe a bluer purple. In virtual or augmented reality environments,interactions by multiple users within the environments may result inobjects, features and/or attributes of the environments changing in acombinational manner based on a synthesis of the individual userinteractions. For example, if the environment represents a game playedby multiple users, then the systems and methods of this disclosure wouldchange the objects, features, and/or attributes of the gamingenvironment based on a synthesis of the user interactions. Suchcombinational interactions may be used to alter the gaming environmentas the game is being played in a manner wholly unexpected by the users.The same type of synthetic changes may be used in other type of systemsincluding image processing, website design, data processing, financialprocessing, tread analyses, etc., where the system integrates all userinteractions to display a combinational analysis of the processingresults. In other embodiments, the systems and methods will capturesensor data relating to the size, proportions, other physical attributesof the users, objects, entities, etc. that are moving within the activezone or zones of the sensor or sensors or sensor array or sensor arrays.

Triggers

In certain embodiments, the systems, apparatuses, and/or interfaces ofthis disclosure and methods implementing them use a marker or animage/character recognition feature to trigger a menu or metadata thatmay then be used with menuing systems of this disclosure or any othermenuing system. These markers or features are similar to a 2D or 3Dbarcode, emoticons, or any object or feature that may be recognized as atrigger. The trigger may be used to unlock certain locked menus or listsfor special access. The triggers may also be used for tailoring triggersto cause the systems, apparatuses, and/or interfaces to invoke specificand pre-defined menus, objects, programs, devices, or other specific orpre-defined items under the control of the systems, apparatuses, and/orinterfaces.

The inventors have found that motion-based eye and/or head trackingapparatuses, interfaces, systems, and methods for implementing them maybe constructed, where the interface is associated with a mobile device,a lap top computer, or a stationary computer or desk top computer havingat least one display device and at least one processing unit and includeat least one eye tracking glasses, head tracking device, or other eyetracking or head tracking device (collectively eye tracking sensor),which may be a sensor associated with the mobile device, the lap topcomputer, or the stationary computer or desk top computer or may be aseparate device in communication with the mobile device, the lap topcomputer, or the stationary computer or desk top computer.

In certain embodiments the interface operates as follows. The at leastone eye tracking sensor senses eye motion or movement as the eye, eyes,or head traverse a plurality of selectable icons or objects displayed onthe at least one display device. The at least one processing unit may beconfigured to receive an output signal(s) from the at least one eyetracking senor as the eye, eyes, and/or head traverse the icons orobjects. The at least one processing unit may also be configured to (1)immediately activate a particular icon or object, when the eye movementmoves in, on, or at the particular icon or object; or (2) activate aparticular icon or object, when the eye movement moves in, on or at theparticular icon or object and stops for a microhold in, on, or at theparticular icon or object. The at least one processing unit may also beconfigured to display a list of attribute icons or object, if theparticular object or icon has associated attributes. The at least oneprocessing unit may also be configured to receive output signal(s) fromthe at least one eye tracking sensor relating to further eye movementtowards a particular attribute icon and to activate the attribute icon.The particular attribute icon may be (1) an adjustable attribute so thatfurther eye movement in one direction will increase the value of theattribute, while eye movement the opposite direction will decrease thevalue of the attribute, (2) the particular attribute icon may beassociated with a sublist, where further eye movement towards as item inthe sublist will activate the item, or (3) the particular attribute iconmay be associated with values set for in a matrix format, where furthereye movement in, on, or at a matric location will set the attributevalue. In certain, embodiments, the at least one processing unit mayalso highlight a particular icon or object, when the eye movement movesin, on, or at the particular icon or object, where the highlighting maybe enlargement or any other highlighting effect the will discriminatethe particular icon or object from the other icons or displays.Additionally, the enlargement may cover non-selected icon or objectsthat are deactivated by the eye movement in, on, or at the particularicon or object.

In other embodiments he interface operates as follows. The at least oneeye tracking sensor senses eye motion or movement as the eye, eyes, orhead traverse a plurality of selectable icons or objects displayed onthe at least one display device. The at least one processing unit may beconfigured to receive an output signal(s) from the at least one eyetracking senor as the eye, eyes, and/or head traverse the icons orobjects. The at least one processing unit may also be configured tohighlight a particular icon or object, when the eye movement moves in,on, or at the particular icon or object. The at least one processingunit may also be configured to (1) receive further eye movement in, on,or at a portion of the highlighted icon or object to confirm theselection of the particular icon or object; and (2) activate theparticular icon or object after eye movement confirmation. Alternately,the at least one processing unit may also be configured to (1) receivefurther eye movement in, on, or at a area or portion of the highlightedicon or object to confirm selection; and (2) to immediately activate theparticular icon or object. In another alternative, the at least oneprocessing unit may also be configured to (1) receive further eyemovement in, on, or at a portion of the highlighted icon or object andstop for a microhold to confirm selection; and (2) activate theparticular icon or object after the microhold confirmation. The at leastone processing unit is also configured to display a list of attributeicons or object, if the particular object or icon has associatedattributes. The at least one processing unit is also configured toreceive output signal(s) from the at least one eye tracking sensorrelating to further eye movement towards a particular attribute icon andto activate the attribute icon and a microhold in, on, or at theparticular attribute icon. The particular attribute icon may be (1) anadjustable attribute so that further eye movement in one direction willincrease the value of the attribute, while eye movement the oppositedirection will decrease the value of the attribute, (2) the particularattribute icon may be associated with a sublist, where further eyemovement towards as item in the sublist will activate the item, or (3)the particular attribute icon may be associated with values set for in amatrix format, where further eye movement in, on, or at a matriclocation will set the attribute value. In certain, embodiments, the atleast one processing unit may also highlight a particular icon orobject, when the eye movement moves in, on, or at the particular icon orobject, where the highlighting may be enlargement or any otherhighlighting effect the will discriminate the particular icon or objectfrom the other icons or displays. Additionally, the enlargement maycover non-selected icon or objects that are deactivated by the eyemovement in, on, or at the particular icon or object.

In other embodiments he interface operates as follows. The at least oneeye tracking sensor senses eye motion or movement as the eye, eyes, orhead traverse a plurality of selectable icons or objects displayed onthe at least one display device. The at least one processing unit may beconfigured to (1) receive an output signal(s) from the at least one eyetracking senor as the eye, eyes, and/or head traverse the icons orobjects; (2) highlight a particular icon or object, when the eyemovement moves in, on, or at the particular icon or object; (3) displaya confirmation icon within the highlighted icon or object, (4) receive amicrohold on the confirmation icon; and (5) activate the highlightedicon. The at least one processing unit is also configured to display alist of attribute icons or object, if the particular object or icon hasassociated attributes. The at least one processing unit is alsoconfigured to receive output signal(s) from the at least one eyetracking sensor relating to further eye movement towards a particularattribute icon and to activate the attribute icon and a microhold in,on, or at the particular attribute icon. The particular attribute iconmay be (1) an adjustable attribute so that further eye movement in onedirection will increase the value of the attribute, while eye movementthe opposite direction will decrease the value of the attribute, (2) theparticular attribute icon may be associated with a sublist, wherefurther eye movement towards as item in the sublist will activate theitem, or (3) the particular attribute icon may be associated with valuesset for in a matrix format, where further eye movement in, on, or at amatric location will set the attribute value. In certain, embodiments,the at least one processing unit may also highlight a particular icon orobject, when the eye movement moves in, on, or at the particular icon orobject, where the highlighting may be enlargement or any otherhighlighting effect the will discriminate the particular icon or objectfrom the other icons or displays. Additionally, the enlargement maycover non-selected icon or objects that are deactivated by the eyemovement in, on, or at the particular icon or object.

In other embodiments, the processing unit may be configured to highlighteach icon or object as the eye movement passes onto, at, into an icon orobject and activate the icon only if the eye movement includes a changein direction within the icon or object screen area, otherwise the iconor object is restored to its normal configuration.

It should be recognized that eye movement that continues beyond thehighlight icon will result in the processing unit removing thehighlighting and await further eye movement. It should also berecognized that the processing unit may be configured to only populatethe display after a threshold movement event occurs or after a correctlogin event occurs, which may be a biokinetic identifier.

Another aspect of the interfaces of this disclosure is that the fasterthe selection object moves toward the pre-selected selectable object orthe group of pre-selected selectable objects, the faster thepre-selected selectable object or the group of preselected selectableobjects move toward the selection object or the faster the objects maymove in a different direction, such as away or at angles to the user'smotion.

Another aspect of the interfaces of this disclosure is that as thepre-selected selectable object or the group of pre-selected selectableobjects move toward the selection object, the pre-selected selectableobject or the group of pre-selected selectable objects or zone or activeareas may also increase in size, change color, become highlighted, haveother effects change, or mixtures or combinations thereof.

Another aspect of the interfaces of this disclosure is that each objectthat has at least one adjustable attribute may include an adjustableactive area associated with each adjustable attribute associated withthe objects that become displayed as the selectable object is augmentedby the motion. Moreover, as the selectable object becomes more certainof selection, the adjustable active areas may increase in size as theselection object moves toward the selectable object or “gravity” pullsthe selectable object toward the selection object. The active areapermits selection to be made prior to any actual contact with theobject, and allows selection to be made merely by moving in thedirection of the desired object. The active area may be thought of as ahalo effect surrounding the object activated by motion toward theobject.

Another aspect of the interfaces of this disclosure is that as motioncontinues, the motion will start to discriminate between members of agroup of pre-selected objects until the motion results in the selectionof a single displayed (discernible) object or a group of displayed(discernible) objects. As the motion continues, the display will beginto discriminate between objects that are aligned with the motion andobjects that are not, emphasizing the selectable objects aligned withthe motion (i.e., objects in the direction of motion) and de-emphasizingthe non-selectable objects not aligned with the motion (i.e., objectsaway from the direction of motion), where the emphasis may be any changein object(s) properties, changes in object(s) positions, or acombination thereof and the de-emphasis may be any change in theobject(s) properties, changes in object(s) positions, or combinationthereof.

Another aspect of the interfaces of this disclosure is the display,movement, and positioning of sublist members or attributes associatedwith object(s) may be simultaneous and synchronous or asynchronous withthe movement and display of the selectable object(s) or displayobject(s) being influenced by the motion of the selection object(s).Once the selection object and a selectable display object touch or theselection object and a selectable display object active area touch orthe selection object and a selectable display object is predicted with athreshold degree of certainty, a triggering threshold event (this may bethe distance of proximity or probability without ever touching), theselectable object(s) is selected and non-selected display object areremoved from the display or fade away or become less prominent or changein such a way that they are recognizable as the non-selected object(s)and the selected object is centered within the display or at apredetermined position, is adjusted to a desired amount if an adjustableattribute, or is executed if the selected object(s) is an attribute orselection command, or any combination of these. If the object is anexecutable object such as taking a photo, launching an application,changing a display orientation, scale or aspect, adjusting the volume,turning on a device, etc., then the execution is simultaneous or acts ina predetermined way with selection. If the object has a submenu, sublistor list of attributes associated with the selected object, then thesubmenu members, sublist members or attributes may become displayed onthe screen is a spaced apart or differentiated format either afterselection or during the selection process, with their distributionbecoming more defined as the selection becomes more and more certain.The same procedure used to select the selected object is then used toselect a member of the submenu, sublist or attribute list. This sameeffect may occur with a combination of executable, submenu, sublist, andlisting attributes. Thus, the interfaces have a gravity or attractive ordistortion or highlighting, or combination of these like action(s) ondisplayed selectable objects. As the selection object moves, it attractsan object or objects in alignment with the direction of the selectionobject's motion pulling those objects toward it, and may simultaneouslyrepel other objects not aligned with the selection object's motion,causing them to move away or be identified as non-selected objects. Thearea, zone or volume of the objects may be simultaneously affected so asto magnify, reduce, or have other effect on the displayed or activearea(s) associated with the object(s). As motion continues or a velocityor acceleration of the motion increase, the pull increases on theobject(s) most aligned with the direction of motion, furtheraccelerating the object toward the selection object until they touch,merge, or cause a triggering selection event, or a combination thereof.If two objects are along the same line or zone, and the closer of thetwo is attracted or selected as motion occurs toward the user, andmotion continues in line, the first object may be treated like anon-wanted object and the second desired object is selected. If motionis stopped, stopped with a predetermined hold or dwell, or slowed to apredetermined threshold amount at the first object, it is consideredselected. If motion continues at the first object, it is considered notselected. The touch, merge or triggering event causes the processingunit to select and activate the object, active an object sublist ormenu, or active an attribute for control, or a combination thereof.

A combination or compound action of these events may occur as well, suchas the eye gaze or eye motion acting as a selection object, and asecondary object of the pointing of a finger or touch on a surfaceacting as another, so the combination provides an enhanced or differenteffect than either one of the objects acting alone. One example inparticular is where eye motion is used in a VR/AR environment tohighlight or cause an effect on a zone of objects, such as expanding anarea most associated with the direction or movement of the eye(s) andhaving objects least probable to diminish or move away from the zonebeing looked at or towards, or displaying a system of objects or menusassociated with the objects (attributes), and a finger direction ofmotion, or a selection object, or an area or volume representing aselectable zone, further defines or selects the selectable objects,attributes or zones. Another example would be to touch a watch orwearable device to begin a selection or scrolling process, while tiltingthe watch further affects the selection, scrolling or attribute process,dynamically affecting the zones and/or objects as described above.Another example would be to move in one direction to adjust thebrightness or volume of a wearable or mobile device with motion of afinger, on or a surface of the device, and having the display expand theaudio zone signifying volume change, then synchronously orasynchronously changing the zonal size of the song choice by moving thedevice in a different direction or axis, or even in the same direction,but being able to differentiate the motion from the motion of thefinger(s), and causing a different scroll, attribute or selection, orcombination of these, to occur more easily based on the relevant zonalsize of motion with what is most relevant to the first motion. In thisexample, using a hand or finger motion above the watch may change thevolume, and rolling the wrist might advance or rewind the song based ondirection of roll, while moving the wrist up or down may play or stop.While beginning to move in the second direction, the probability ofchanging the song increases as more motion is applied, and the attributeof adjusting volume diminishes until it is non-selectable while changingthe song.

Methods

Embodiments of this disclosure provide methods for implementing theselection protocol using the user interfaces of this disclosure. Themethods include selecting and activating selectable objects, selectingand activating members of a selectable list of virtual and/or realobjects, selecting and activating selectable attributes associated withthe objects, selecting and activating and adjusting selectableattributes, zones, areas, or combinations thereof, where the interfacesinclude at least one display or other user feedback unit, at least onemotion sensor (or data received therefrom), and at least one processingunit in communication with the user feedback units and the motionsensors or motion sensor data. The interfaces also may include powersupplies, battery backups, and communications software and hardware forremote control and/or remote monitoring. The methods include sensingmotion or movement sensed by the motion sensor(s), generating an outputsignal and sending the output signal to the processing unit. The methodsalso include converting the output signal into a command output via theprocessing unit. The command output may be a start command, whichactivates the feedback unit or activates the feedback unit and generatesat least one selection or cursor object or activates the feedback unitand generates at least one selectable object or activates the feedbackunit and generates at least one selection or cursor object and at leastone selectable object. The selection object may be discernible or not(displayed or not). The motion may be generated by an animal or bodypart or parts, a human or body part or parts, a machine, or a real worldobject under control of an animal, a human, or a robot or roboticsystem, especially when the motion being sensed is within a 3D activesensing volume or zone. Once activated, the methods monitor sensedmotion or movement within the active zone(s) of the motion sensor(s),which is used to move the selection object on or within the userfeedback unit in accord with the motion properties (direction, velocity,acceleration, and changes of one or more of these properties) toward aselectable object or a group of selectable objects or a pre-selectedobject or a group of pre-selected objects, zones or areas. At the sametime, the methods either move the non-selected objects away from theselection object(s), cause the non-selected object to fade, disappear orother change other properties of the non-selected objects, orcombinations thereof. The pre-selected object or the group ofpre-selected objects are the selectable object(s) that are most closelyaligned with the direction of motion of the selection object.

Another aspect of the methods of this disclosure is that movementtowards an executable area, such as a close/expand/maximize/minimizefunction area(s) or object(s) of a software window in an upper rightcorner may cause an executable function(s) to occur, such as causing theobject(s) to expand or move apart so as to provide more space betweenthem and to make it easier to select each individual object or a groupof objects.

Another aspect of the methods of this disclosure include interfaces isthat object selection or menu selection may be grouped together suchthat as movement is made towards a group of objects, the group ofobjects simultaneous rearrange themselves so as to make individualobject selection or menu selection easier, including moving arcuately orto corners of a designated area so as to make discrimination of thedesired selection easier.

Another aspect of the interface is that proximity to the selectionobject may cause the selectable objects, zones or areas most alignedwith the properties of the sensed motion to expand, separate, orotherwise move in such a way so as to make object discrimination easier,which in turn may cause associated subobjects or submenus to be able tobe selected by moving the subobjects or submenus towards the selectionobject. Additionally, they could be selected or activated by moving intoan active area designated by distance, area or volume from or aroundsuch objects, thereby selecting the object functions, menus orsubobjects or submenus. The movement or attribute change of thesubobjects or submenus may occur synchronously or asynchronously withthe movement of the primary object(s).

Another aspect of the interfaces is that the faster the selection objectmoves toward the pre-selected object or the group of preselectedobjects, the faster the pre-selected object or the group of preselectedobjects move toward the selection object(s), and/or the faster theunselected objects may move away from the selection object(s). Ofcourse, any differentiation of direction may be used, where theunselected objects move towards the user and the preselected objectsmove away.

Another aspect of the interfaces is that as the pre-selected (meaningthe objects that are most closely aligned with the properties of themotion) object or the group of pre-selected objects move toward theselection object, the pre-selected object or the group of pre-selectedobjects may either increase in size, change color, become highlighted,change some other effect, change some characteristic or attribute, or acombination thereof. These same, similar or opposite changes may occurto the unselected objects or unselected group of objects. Another aspectis that, based upon a user's previous choices, habits, motions orpredicted motions, the attributes of the objects may be changed suchthat they move faster, increase in size or zone, or change in such a waythat the object with the highest percentage of user intent is theeasiest and most likely to be selected as shown in FIGS. 4A-D anddescribed in the associated text below.

Another aspect of the interfaces is that as motion continues, the motionwill start to discriminate between members of the group of pre-selectedobject until the motion results in the selection of a single selectableor displayed object or a single group of selectable objects, or zone orgroup of zones, or any combination(s) of these. Once the selectionobject and a selectable object active area touch or the selection objectand a selectable display object is predicted with a threshold degree ofcertainty, a triggering threshold event (this may be the distance ofproximity or probability without ever touching), the selectable objectis selected and non-selected object are removed from the display or fadeaway or become less prominent or change in such a way that they arerecognizable as non-selected object(s). Once selected, the selectedobject may become centered within the display or at a predeterminedposition within the display. If the selected object has a singleadjustable attribute, then motion may adjust the attribute a desired orpre-defined amount. If the selected object is executable, then theselected object is invoked. If the selected object is an attribute orselection command, then the attribute may be adjusted by additionalmotion or the selection may invoke a command function. Of course, thesystems may do all or any combination of these processes. If the objectis an executable object such as taking a photo, turning on a device,etc., then the execution is simultaneous or acts in a predetermined waywith the selection. If the object is a submenu, sublist or list ofattributes associated with the selected object, then the submenumembers, sublist members or attributes are displayed on the screen in aspaced apart format or appear as the selection becomes more certain andthen persist once selection is certain or confirmed, and may bedisplayed at one or more levels, and any or all synchronously,asynchronously or simultaneously. The same procedure used to select theselected object is then used to select a member of the submenu, a memberof the sublist or a particular attribute. Thus, the interfaces have agravity like action on displayed selectable objects that move themtoward the selection objection as certainty increases. As the selectionobject moves, it attracts an object or objects in alignment or relationwith the properties of the sensed motions (direction, speed,acceleration, or changes in any of these primary properties) of theselection object pulling the object(s) meeting this criterion toward theselection object. Simultaneously, synchronously or asynchronously,submenus or subobjects may become visible if they were not so to beginwith and may also move or change in relation to the movement or changesof the selected objects. Simultaneously, synchronously, orasynchronously, the non-selected objects may move or change away fromthe selection object(s). As motion continues, the pull increases on theobject most aligned with the properties (e.g., direction) of motion ormovement, further moving or accelerating the object toward the selectionobject until they touch, merge, or reach a triggering event close enoughto touch an active area or to predicted the selection to a thresholdcertainty. The touch, merge, or triggering event causes the processingunit to select and activate the object. The object(s) may also bedefined as an area in between objects, giving a gate-like effect toprovide selection of sub-menu or sub-objects that are aligned with themotion of the selection object and are located between, behind, or atthe same angle but a different distance than this gate. Furthermore, aback object or area may be incorporated to undo or reverse effects orchanges or motions that have occurred to objects, whether selectable ornot.

Embodiments of this disclosure provide Embodiments of this disclosurerelate to systems and methods implements on a processing unit ordistributed network of processing units, where the systems include atleast one processing unit, at least one motion sensor, at least one userinterface, and dynamic environment software and methods include softwaresteps to implement the software systems on the processing units, wherethe dynamic environment software produces dynamic environments forobject and attribute display, manipulation, and/or interaction.

More particularly, this disclosure relate to systems and methodsimplements on a processing unit or distributed network of processingunits, where the systems include at least one processing unit, at leastone motion sensor or at least one sensor output, at least one userinterface, and dynamic environment software and methods include softwaresteps to implement the software systems on the processing units, wherethe dynamic environment software produces dynamic environments forobject and attribute display, manipulation, and/or interaction, wherethe dynamic environment software produces dynamic environments forobject and attribute display, manipulation, and/or interaction, wherethe dynamic environments are produced by the dynamic environmentsoftware and include a plurality of objects and associated attributes sothat the objects and/or attributes are highlighted and/or differentiatedone from the other, where the highlighting may evidence priority,directionality, content, type, activation procedures, activationparameters, control features, or other properties that are associatedwith the objects and/or attributes and motion sensed by the motionsensors in electronic communication with the processing units permitmanipulation and/or interaction with the elements of the dynamicenvironments either causing the environment to change in response to thesensed motion or causing object and/or attribution selection and/oractivation.

Control Constructs Viewable Through Eye Tracking Glasses

Embodiments of this disclosure broadly relate to methods includingdisplaying, via a processing unit, a virtual reality (VR), augmentedreality (AR), or a mixed reality (MR) environment including a pluralityof objects, a plurality of scenes, and/or a plurality of attributesassociated with the environment and/or objects or scenes on a displaydevice in communication with processing unit. The methods also includeviewing, via the processing unit, the environment through eye trackingglasses in communication with the processing unit and superimposing, inor on the environment via the processing unit, an interactive controlconstruct including a plurality active control areas. The methods alsoinclude displaying, via the processing unit, a selection object forvisualizing eye movement from the glasses in the environment or in or onthe control construct, sensing eye movement via the glasses and movingthe selection object correspondingly, and if the movement is on or in aparticular active control area of the construct, then converting themovement into a construct function associated with the particular activecontrol area of the construct or if the movement is on or in theenvironment, then converting the movement into a selection and/oractivation function of a particular object or a selection, activation,and/or attribute function of the environment.

In certain embodiments, the methods further include repeating thesensing and if steps, until an exit or stop command is selected. Inother embodiments, the methods also include prior to the environmentdisplaying step, sensing a first eye movement, wherein the firstmovement includes at least one property exceeding a threshold value. Inother embodiments, the control areas comprise a fast forward movementarea, a no movement area, a backward movement area, a pan left area, apan right area, and a no pan area, wherein each area has a correspondingmovement function or a corresponding panning function associatedtherewith. In other embodiments, the control areas comprise a pluralityof bands, each of the bands having a plurality of regions. In otherembodiments, the bands comprise a fast forward band, a slow forwardband, a stop or no motion-band, a fast backward band, and a slow forwardband, wherein each band has a corresponding movement function associatedtherewith, and the regions comprise a pan left region, a pan rightregion and a no pan region interposed therebetween, wherein each bandhas a corresponding panning function associated therewith. In certainembodiments, the methods further include sensing eye movement via theglasses and moving the selection object correspondingly, if the movementis on or in a particular band and a particular region of the particularband, then converting the movement into the corresponding movementfunction and panning function and moving through the environmentaccordingly or if the movement is on or in the environment, thenconverting the movement into a selection and/or activation function of aparticular object or a selection, activation, and/or attribute functionof the environment. In certain embodiments, the methods further includeprior to the if steps, confirming the selection of the particular areaof the construct, or confirming the selection of the object or attributein the environment.

Embodiments of this disclosure broadly relate to apparatuses including adisplay device, an input device, eye tracking glasses, and a processorin communication with the display device, the glasses and the inputdevice. The processor is configured to (a) display a virtual reality(VR), augmented reality (AR), or a mixed reality (MR) environmentincluding a plurality of objects, a plurality of scenes, and/or aplurality of attributes associated with the environment, objects and/orscenes on the display device, (b) initiate the glasses to view theenvironment, (c) superimpose, in or on the environment, an interactivecontrol construct including a plurality active control areas viewablethrough the glasses, (c) display a selection object for visualizing eyemovement from the glasses in or on the environment or in or on thecontrol construct, (d) sense eye movement via the glasses and moving theselection object correspondingly, and (e) if the movement is on or in aparticular active control area of the construct, then converting themovement into a function associated with the particular active controlarea of the construct; or if the movement is on or in the environment,then converting the movement into a selection and/or activation functionof a particular object or a selection, activation, and/or attributeadjustment function of a particular attribute of the particular objector of the environment.

In certain embodiments, the process is further configured to prior tothe environment display, sense a first movement from the glasses or fromthe input device, wherein the first movement includes at least onemovement property exceeding a threshold value. In other embodiments, thedevice comprises a mobile device including the display device, the inputdevice, and the processing unit. In other embodiments, the devicecomprises a mobile device including the display device, the inputdevice, and the processing unit. In other embodiments, the displaydevice comprises a touchscreen and the input device is a motion sensor.In other embodiments, the mobile device comprises a cell phone, awearable device, a tablet computer, or a laptop computer.

Eye Tracking Methods and Apparatuses for Controlling Icons

Embodiments of this disclosure broadly relate to methods includingdisplaying, via a processing unit, a plurality of icon on a displaydevice in communication with the processing unit. The methods alsoinclude sensing, via the processing unit, eye movement via eye trackingglasses in communication with the processing unit, where the movement isdiscernibly towards a particular icon and expanding, via the processingunit, the particular icon. The methods also include activating, via theprocessing unit, the particular icon once the movement enters theparticular icon and displaying, via the processing unit, one or aplurality of attribute icons associated with the particular icon. Themethods also include sensing, via the processing unit, further eyemovement via the glasses discernibly towards a particular attributeicon, selecting and activating the particular attribute icon and if theparticular attribute icon is adjustable, sensing further eye movementvia the glasses to adjust the attribute value.

In certain embodiments, the methods further include repeating the steps.In other embodiments, the methods further include prior to thedisplaying icon step, sensing a first eye movement, wherein the firstmovement includes at least one property exceeding a threshold value. Inother embodiments, the expanding step further includes displaying aconfirmation icon within the expanded particular icon and the activatingstep comprises sensing eye movement via the glasses towards theconfirmation icon and activating the particular icon when the movementstop for a microhold over the confirmation icon. In other embodiments,the activating step further includes a microhold over the particularicon. In other embodiments, the selecting and activating the particularattribute icon step further includes a microhold over the particularattribute icon. In other embodiments, the selecting and activating theparticular attribute icon step further includes expanding the particularattribute icon and displaying a confirmation icon within an expandedparticular icon and the activating step comprises sensing eye movementvia the glasses towards the confirmation icon, selecting and activatingthe particular attribute icon, and if the particular attribute icon isadjustable, sensing further eye movement via the glasses to adjust theattribute value.

Embodiments of this disclosure broadly relate to apparatuses including adisplay device, an input device, eye tracking glasses, and a processor,in communication with the display device, the glasses and the inputdevice. The processor is configured to (a) display a plurality of iconon a display device in communication with the processing unit, (b) senseeye movement via eye tracking glasses, where the movement is discerniblytowards a particular icon, (c) expand the particular icon, (d) activatethe particular icon once the movement enters the particular icon, (e)display one or a plurality of attribute icons associated with theparticular icon, (f) sense further eye movement via the glassesdiscernibly towards a particular attribute icon, (g) select and activatethe particular attribute icon, (h) if the particular attribute icon isadjustable, sense further eye movement via the glasses to adjust theattribute value.

In certain embodiments, the processor is further configured to repeatthe display, sense, expand, activate, display, select and activate andattribute adjust. In other embodiments, the processor is furtherconfigured to prior to the display icons, sense a first eye movement,wherein the first movement includes at least one property exceeding athreshold value. In other embodiments, the expand further includesdisplay of a confirmation icon within the expanded particular icon andthe activate comprises sense eye movement via the glasses towards theconfirmation icon and activate the particular icon when the movementstop for a microhold over the confirmation icon. In other embodiments,the activate further includes a microhold over the particular icon. Inother embodiments, the select and activate the particular attribute iconstep further includes a microhold over the particular attribute icon. Inother embodiments, In other embodiments, the select and activate theparticular attribute icon step further includes expand the particularattribute icon and display a confirmation icon within an expandedparticular icon, and the activate comprises sense eye movement via theglasses towards the confirmation icon, select and activate theparticular attribute icon, and if the particular attribute icon isadjustable, sense further eye movement via the glasses to adjust theattribute value.

Training Methods and Apparatuses Using Computer Generated Constructs

Embodiments of this disclosure broadly relate to methods including (1)capturing, via a processing unit, trainee images or a trainee imagesequence and trainer images and/or a trainer image sequence from atleast one motion sensor corresponding to a specific task; (2)constructing, via the processing unit, constructs of the trainee imagesor the trainee image sequence and constructs of the trainer images orthe trainer image sequence on an image by image basis and on a featureby feature basis; (3) scaling, via the processing unit, the traineeand/or trainer constructs on an image by image and on a feature byfeature basis so that the constructs are spatially and temporallyconformed; (4) comparing, via the processing unit, the scaled constructson an image by image basis and on a feature by feature basis; (5)determining, via the processing unit, differences between the scaledconstructs on an image by image basis and on a feature by feature basis;(6) overlaying, via the processing unit, the scaled trainee constructonto the scaled trainer construct on an image by image basis; and (7)displaying, via the processing unit, the overlaid constructs on an imageby image basis on a display device in communication with the processingunit.

In certain embodiments, the methods further include (8) highlighting,via the processing unit, the differences on an image by image basis andon a feature by feature basis on the display device. In otherembodiments, the methods further include (9) repeating the steps (1)-(7)until a stop function is invoked to stop the training exercise or thedifferences satisfy at least one completion criteria. In otherembodiments, the at least one completion criteria comprises conformityor substantially conformity between the trainee scaled construct and thetrainer scaled construct on an image by image basis or the at least onecompletion criteria comprise minimizing one, some or all of differencesas evidenced by changes in the differences upon repeating the exercisebeing less than a minimal value, where the minimal value is less than orequal to a 5%, less than or equal to a 2.5%, less than or equal to a 2%,or less than or equal to a 1%. In other embodiments, the start and stopfunctions may be activated by motion-based processing, gesture basedprocessing or a hard select processing.

Embodiments of this disclosure broadly relate to apparatuses including adisplay device, an image capture device, an input device, an interface,and a processor. The interface is configured to capture trainee imagesor a trainee image sequence from the image capture device and trainerimages or a trainer image sequence from the image capture device andreceive input from the input device to invoke a start function or a stopfunction. The processor is configured to construct a trainee image byimage construct and a trainer image by image construct; scale thetrainee image by image construct and/or trainer image by image constructso that the constructs are conformed spatially and temporally; comparethe scaled constructs on an image by image basis; determine differencesbetween the scaled constructs on an image by image basis; overlay thescaled trainee construct onto the scaled trainer construct on an imageby image basis; and display the overlaid constructs on an image by imagebasis.

In certain embodiments, the processor is further configured to highlightthe differences on an image by image basis and a feature by featurebasis. In other embodiments, the processor is further configured torepeat the construct, scale, compare, determine, overlay and displayevents until the a stop function is invoked to stop the trainingexercise or the differences satisfy at least one completion criteria. Inother embodiments, the at least one completion criteria comprisesconformity or substantially conformity between the trainee scaledconstruct and the trainer scaled construct on an image by image basis orthe at least one completion criteria comprise minimizing one, some orall of differences as evidenced by changes in the differences uponrepeating the exercise being less than a minimal value, where theminimal value is less than or equal to a 5%, less than or equal to a2.5%, less than or equal to a 2%, or less than or equal to a 1%. Inother embodiments, the start and stop functions may be activated bymotion-based processing, gesture based processing or a hard selectprocessing.

Information and Communications System

360IVX Training

Quantum Interface has created a hands-free AR and VR-based training,service, logistics and awareness tool to associated interactive contentto any point is virtual or real space. It is based on our patented useof motion interactions, so content and controls can be added to anyenvironment, augmented, interacted with, and analyzed in real time forthe user confidence while interacting. It may be deployed across anyhardware or software with the same User Interaction, providing aconsistent Universal Interface every time, no matter what is used(Cardboard, PC, VR, AR, XR, touch, mobile, etc.). This enables existingmediums, such as 360 videos, to become entire training systems, digitallibraries and curricula, and the same builds may be transferred to ARand touch environment. We make it easy to use, so a non-programmer canget immediate results through an easy to use interface. The embedded QIinteraction technology provides the fastest, most natural experiencewith the highest cognition and retention available. Protected by US andinternational patents and patents pending.

There is a need for a new way to communicate information for training,logistics, servicing (like in, assessment and awareness (such assituational awareness). The needs may be separate or combined. Theproblem with current systems is they do not provide the deeperunderstanding necessary to know things comprehensively, and the contentis not presented in a way that the brain has the maximum cognition andretention.

An example of this is with a training system. In trying to meet the needfor students to learn a process as fast as possible, different formatsare used. Text, audio, video, 360 video (VR), simulated training,hands-on, etc. are used. The students then combines all these differentformats and associated information in their heads. What is missing isthe ability to combine these different formats together so they can beseen, heard and interacted with simultaneously, or in one location.

In a previous application, we have described “Hot spots” or interactivepoints where content may be added, and multiple layers of it, to anypoint in space (virtual or real). An example is that one may take adocument or image and attach videos, images, XR experiences, audio,hyperlinks, etc.

In an earlier application, we disclosed performing cloud services in thecloud where the interactions would occur on a device, but calculationsfor the processes and other aspects could be performed in the “Cloud”—asystem of servers, CPUs and data storage groups that may be accessed viathe web or other methods so content and processing may be accessed fromany location with data communication abilities.

In certain embodiments, the virtual training systems, apparatuses,and/or interfaces and methods for implementing them, may include any,some, or all of the following devices and/or abilities set forth below.

Devices

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include devices capable ofinteracting with a user such as motion sensing devices (e.g., cameras,accelerometer devices, joy sticks, ball controllers, mouses, touchscreens, etc.), audible sensing devices (e.g., microphones, etc.),haptic devices, smell devices, body tracking devices (e.g., eye trackingdevices, head tracking devices, VR goggle devices, VR body suit devices,glove devices, etc.), any other devices capable of generating outputdata derived from user activity, or any combination of these devices.

Interaction Points or Hot Spots

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability toattach or associate interaction points or hot spots to one or moreplaces or locations within a VR, AR, MR, and/or XR environment, whereinthe hot spots may be pre-defined and populated at places or locationsaccording to a protocol implemented by the virtual training systems,apparatuses, and/or interfaces and methods for implementing them, atstart up and/or during different training aspects or sessions. Forexample, if the training is for air craft, then the virtual trainingsystems, apparatuses, and/or interfaces and methods for implementingthem, may associated hot spots all equipment in the air craft, withvarious features, properties, and/or structures in the sky (clouds,winds, sheers, storms, other planes, targets, etc.) and/or ground(mountains, valleys, other features and/or properties, building and/orother man-made structures, flight control centers, highways, targets,etc.). For example, in the case of videos or other streaming media (timebased), then the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, may associated hot spotswith any feature and/or property in the video or streaming media at anytime during the video or media, wherein the hot spots may change withthe nature of each scene in the video, and/or each training segment orsession, and/or based on certain criteria that a user may activate whileplaying the video or media.

Single Input or Multiple Input Processing

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability to usevarious input data, independently or collectively, to control,manipulate, activate, adjust, alter, and/or modify any controllablefeature, property, characteristic, and/or attribute of the environment,wherein the input data may come from one or more of the devices setforth above.

Tools to Create, Edit, and/or Augment Environmental Content

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability and/ortools to create, delete, edit, alter, modify, augment, etc. content,media, controls, hot spots, adjustable attributes, features, structures,properties, attributes, characteristics, etc. associated with theenvironment.

Content Uploading or Downloading

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability toupload and/or download content to the cloud and/or another remotestorage location.

Cloud Based Computing

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability usethe tools that activate or deactivate various input devices locallyand/or input devices resident in the cloud and/or other remote storagelocation.

Input Data Types

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability tointeract with the environment using body part input data (e.g., eyes,head, fingers, hand, etc.), using touch data, using touchless or motiondata, audio data, using visual data, using audiovisual data, usingcontroller data, using remote controller data, using robot or roboticsdata, and/or any other user input data.

Real-time Interaction Confidence, Cognition, And/or Retention

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability tomeasure real-time confidence, cognition, and/or retention through theinterface provided on the devices or via the web.

Use of Analytics

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability to useanalytics (AI, calculations, functions or other ways of providingmetrics).

Use of Analytics for Determining Competencies Assessments

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability totake those metrics (real-time and/or cloud AI) and use by themselves orworking together to provide metrics for user competencies, testing ofany kind, and assessment of awareness and cognition, etc.

Generating Reports

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability togenerate reports and/or other reporting measures.

Using Metrics to Recommend Characteristics of Training

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability totake the metrics and suggest to trainers or trainee subjects orcharacteristics that the trainee may need more training and suggesttechniques that the trainee may use to improve the trainees competenciesin the subject or characteristic such as providing more or betterinformation and/or the communication of information concerning thesubject or characteristic needing improvement or suggesting an overlayconstruct to assist the trainee mimic a trainers actions.

Automating the Training

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability tocollect data and analytics sufficient to automate any given trainingexercise and to incorporate the automated training exercise into thevirtual training systems, apparatuses, and/or interfaces and methods forimplementing them.

Integrating the Virtual Training with Other Programs

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability tointegrate scheduling abilities and/or other types of systems (such ascontact managers, project management systems, calling systems, financialsystems, robotics, machines, hardware, etc., and any other type ofsystems/software/hardware).

Integrating Security Measure or Systems

In certain embodiment, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, include the ability tointegrate security measures, including profiles, signature types,passwords, etc. at any point and in combination with the system.

360° Environments

For example, the virtual training systems, apparatuses, and/orinterfaces and methods for implementing them, have the capability ofinputting or accessing a 360-degree video (360° video) and adding orassociating one or more interactive points or hot spots to or with oneor more location, objects, structures, features, attributes, properties,and/or characteristics featured in the 360° video. The interactivepoints or hot spots may be accessible a specific time in the 360° video,may be accessible for a specific period of time in the 360° video, ormay be accessible during the entire 360° video. Additionally, theinteractive points or hot spots accessible at any given time or timeperiod during the 360° video may be the same or different and may changedepending on the level of training. For example, as the level oftraining increases, the number of interactive points or hot spots mayincrease and the information associated with them may also increase. The360° video may show a way to do something, but by associatinginteractive points or hot spots that include information at differentlocations in the 360° video and/or different point during the 360°video, the trainee may experience improved cognition and retention ofthe training material. For example, while looking at an instrument on acontrol panel in a 360° video, the training systems may display aninteractive point or hot spot (which may be a selectable object)associated with the instrument. If the interactive point or hot spot orselectable object includes a single information item (e.g., text file),then looking at the interactive point or hot spot or selectable objector otherwise activating the interactive point or hot spot or selectableobject will display the text file. If the interactive point or hot spotor selectable object includes more than one item, then looking at theinteractive point or hot spot or selectable object, a list of selectableinformation objects would appear and the trainee may use any scrolling,selecting and activating procedure available in to the systems to selectand activate a particular selectable information object and to view theinformation, which may be textual, audio, visual (video), audiovisual,tactile, etc. Thus, the systems may associate interactive points or hotspots or selectable objects with all instruments of the control panel sothat the trainee is able to obtain information about each instrumentassociated with the control panel. Additionally, the systems may changewhich of the interactive points or hot spots or selectable objects arevisible and accessible at different times during the 360° video. Theinformation associated with an instrument interactive point or hot spotor selectable object may be an animated construct to show how theinstrument works and how it is related to the entire control panel orthe entire contents of the 360° video. Also, the text form a resourcemanual may be displayed Animated or static lines (vectors) may be drawnbetween the elements to establish the relationship between them all andas the user interacts, using Qi technology, the objects may respondpredictively and with a priority to further establish the correctcognition and retention patterns. Testing may be incorporated to assessconfidence while interacting, what requirements are met, and what elseneeds to be learned and/or accomplished. If metrics do not meet athreshold, the User may be taken back to a place in the system toreview, or more content may be introduced. Once an interactive learningenvironment section is complete, the video may resume and until the nextsection.

Interactions may also occur while the video continues to play. Severalinstances may play at once, where the user can move through instances ofthe same thing. For instance, a user may be able to see the same eventseach day of the week, and move through the days while the video isplaying. The interaction in this case may be a UI function (Select,Activate, Scroll, Attribute Control), or may be a Hot spot that providesAttribute control and content navigation.

This system would typically work by a person taking a video anduploading it to a local device or to the cloud, or other repository.Other content, such as 3D objects, images, text, video, VR and ARcontent, etc. can reside in a local location, or on the cloud (web,online repository, remote server, etc.).

By using a set of tools that can reside on a device or on-line, the usercan take a video and augment it by adding Interactive zones (hot spots)to it at any location, area, volume and/or time in the video. This Hotspot may contain one or more interactive attributes, control attributes,menu design attributes, content design attributes, animation attributes,time attributes, relational attributes, associative attributes, or anyother kind of attributes, and may provide these individually,simultaneously or in any configured way.

The content associated with these points may then be associated with theinteractive attributes. This augmented content may then be installed ona device through download, wirelessly, through a wired connection, orany other distribution means (the preferred environment is to downloadthe experience wirelessly). The experience may also be “streamed” so theinstallation may be minimal or at any level, the rest of the experiencebeing performed via the web, cloud, or other experiential medium.

As the user is in the experience, as the hot spots become interactivethe video pauses until the extra layers or content is moved through asdesigned. These interactions may also occur while the video continues oris slowed down, or even reversed to “re-live” the experience again afterinitiating.

The experience may then be analyzed as it is being used (using Qivector-based confidence metrics), providing metrics and analysis thatmay be used for reports, logic-based (AI) responses, modifications orstimuli, and overall analytics. This may also be combined withafter-the-fact metrics, and analysis performed on the device or in thecloud, so AI, logic, or any type of analytics may be combined withreal-time confidence metrics to assess the status of the user, determinecompetency, and or remedial or repletion needed to provide the highestcognition and retention.

Changes may be made while in the environment, from the device, exitingfrom the “play” program, in real-time as the environment is beingexperienced

This may then be used for further enhancements or changes.

The result may be stored for further use later, and accessed on-demandfrom anywhere, on any device.

The environment may be a multi-user experience as well, where others maysimultaneously experience, augment, or affect the outcome.

Video or other overall environmental controls may be incorporated intothe experience.

The preferred interactions are through gaze-based dynamic motions(motion of the eyes/head or both), but may include using controllers,remotes, voices, audio utterances, gestures, touch surfaces or otherinputs, singularly or in combination.

Though the preferred experience environment is a headworn device (ormobile device that can be used as a headworn), the same training,servicing, awareness, etc. and interactions can occur on a mobile device(phone, tablet, watch, etc.) or on a PC with cursor interactions, or anyother environment. It may be also experienced with a projected displaythat is 360, 180 (or any degree), flat, holographic, or any otherenvironment that may provide interactivity with content.

A base understanding of why this system is better is to realize how thebrain learns. Our brain provides location and time data to things weremember well. When we have more time and positional stamps associatedwith a process or concept, it strengthens the relationship between usand that information. By causing the objects to respond as we move in apath, we create large amounts of touch points along that path,increasing the impression on our brain. Another advantage to usingdynamic interactions, where objects respond to our motions as we move,in real time, is that it causes our brain to be alert at a higher levelas we perceive these objects are now intelligent. The GUI system usedtoday does not allow the objects to respond as we move to them, onlyafter we have reached them. Our brain perceives these objects asnon-sentient, and therefore we are not on the higher alert status.

By showing multiple levels of objects at once, our foveated and nearperipheral view is engaged, activating the middle areas of our brain,which are the fastest processing areas. At the same time, showing therelationships between multiple objects and/or levels at once displaysmore spatiality to the concepts, activating the same part of the brainthat groups elements together to see patterns and understand things,which causes higher cognition and retention.

The system described further above includes all these interactions,design elements and concepts. When all these are put together, thespeed, quality and benefits of the system are maximized.

This is not only good for training, but even awareness, so that in AR.MR, VR, and/or XR, where the hands are being used for a task, or in VRwhere a battlefield assessment is needed, these same Hot spots andinteraction designs are used for navigation through menus and content,and the same cognitive and retention quality is provided. When usingthis system to create, use, modify, analyze and improve any environmentor task, the benefits are heightened awareness, faster and heightenedcognition, better retention, and a more natural way to gather, processand use information and controls.

Specific Embodiments

Embodiments of this disclosure broadly relate to methods includingcapturing trainer images or a trainer image sequence corresponding tothe trainer performing a task, routine, or program, and constructing acomputer generated trainer constructs on an image by image basiscorresponding to the trainer performing the task, routine, or program.The methods also include capturing trainee images or a trainee imagesequence corresponding to the trainee performing the task, routine, orprogram and constructing a computer generated trainee constructs on animage by image basis corresponding to the trainee performing the task,routine, or program. The methods also include scaling the traineeconstructs and trainer constructs on an image by image basis, comparingthe scaled trainee constructs and trainer constructs on an image byimage basis and on a feature by feature basis, and determiningdifferences between the trainer and trainee features on an image byimage basis. The method also includes overlaying the scaled traineeconstructs onto the scaled trainer constructs on an image by image basisand displaying the overlaid scaled constructs. The displaying occurs asthe trainer or trainee performs the task, routine, or program or anypart thereof. The method also includes differences testing thedifferences between the trainer performance and the trainee performanceon an image by image basis and feature by feature basis, if each of thedifferences exceed the one or more minimum difference criteria, thencontrol is transferred to the trainee capturing step so that the traineerepeats the task, routine, or program or any part thereof, and if eachof the difference is less than or equal to the one or more minimumdifference criteria, then stopping. It should be recognized that thesteps may be performed in different sequences.

In certain embodiments, the methods further comprise highlighting thedifferences according to a highlighting format, wherein the highlightingformat includes color, shade, pulsating, or any highlighting technique.In other embodiments, the methods further comprise illustrating how thetrainee may adjust body and/or body part positioning in each image orover a sequence of images so that the trainee sees how to adjust bodyand/or body part positioning in each image or over a sequence of imagesto minimize the highlighted differences. In other embodiments, therepeating steps continues until a stop function is invoked using aninput device or the differences are less than or equal to the one ormore difference criteria. In other embodiments, the minimal differencevalues are less than or equal to a 5%, less than or equal to a 2.5%,less than or equal to a 2%, or less than or equal to a 1%. In otherembodiments, the repeating step comprises the trainee repeats the entiretask, routine or program or any part of the task, routine or program orthe repeating step comprises suggested parts of the task, routine orprogram to focus on those parts of the task, routine or program needingthe most correction. In other embodiments, the repeating step comprisescustomized parts of the task, routine or program customized to eachtrainee based on trainee learning proclivities. In other embodiments,the constructs comprise a polygon construct, where all features of theimages are represented by a plurality of polygons capable of beingscaled so that the trainee image features are conformable to the trainerimage features. In other embodiments, the constructs comprise atessellated construct capable of being scaled so that the trainee imagefeatures are conformable to the trainer image features. In otherembodiments, the constructs are a finite element constructs capable ofbeing scaled so that the trainee image features are conformable to thetrainer image features. In other embodiments, the scaling comprisesspatial and temporal scaling so that the trainee image features areconformable to the trainer image features on a continuous or discretebasis. In other embodiments, the differences comprise whole bodyposition differences, body part position differences, body positiontemporal differences, body part position temporal differences, any otherfeature differences, or any combination thereof on an image by imagebasis, a sequence of images basis or an entire image sequence basis. Inother embodiments, the image capturing comprises capturing images on anytime frame from a continuous capture or a time increment capture,wherein the time frame is between about 1 microsecond and a 10 secondsor sub time frame value.

Embodiments of this disclosure broadly relate to methods includingidentifying a non-animated training task, routine, or program,developing a VR or AR/MR/XR training task, routine, or programcorresponding the non-animated task, routine, or program, generatinginformation about (a) important, relevant, meaningful, and/orinteresting features, attributes, properties, and/or characteristicsassociated with the training task, routine, or program and (b)important, relevant, meaningful, and/or interesting features,attributes, properties, and/or characteristics associated with equipmentor devices used in the training task, routine, or program, gatheringinformation about (a) important, relevant, meaningful, and/orinteresting features, attributes, properties, and/or characteristicsassociated with the training routine and (b) important, relevant,meaningful, and/or interesting features, attributes, properties, and/orcharacteristics associated with equipment or devices used in thetraining task, routine or program, storing the generated or gatheredinformation in local databases and/or remote databases. The methods alsoinclude analyzing the task, routine, or program, generating a VRenvironment or a VR/AR/MR environment for the non-animated trainingtask, routine, or program, and associating/populating hot spotscomprising the generated and/or gathered information with (a) eachimportant, relevant, meaningful, or interesting feature, attribute,property, and/or characteristic of the task, routine, or program and (b)each important, relevant, meaningful, and/or interesting feature,attribute, property, and/or characteristic of the equipment and/ordevices associated with the task, routine, or program. The methods alsoinclude collecting/capturing expert data as an expert performs the VR orthe AR/MR/XR training task, routine, or program, processing the expertperformance data, overall data and task data and storing the data in thesystem databases. The methods also include providing the expert withoverall performance data and/or part performance data to gain insightfrom the expert on how to improve the training routine and environment,collecting/capturing trainee data as a trainee goes through the trainingroutine in the training environment, wherein the data includes data onhow the trainee accomplished each training task, data on what hot spotsthe trainee activates, data on the duration of time the trainee spendsin teach hot spot and data on the type of information viewed, dataindicating a level of competency in each task, data indicating anoverall competency, and data associated with ease of using each task,ease of using the entire training routine, and each of using andinteracting with the environment, and providing feedback to the traineeand/or to the supervisors before, during, and/or after each task orbefore, during, and/or after the entire training routine to the trainee.The methods also include collecting and storing all feed back data andevaluation data made by supervisors, collecting/capturing and storingthe feedback data to the system databases. The methods also includeinforming the trainee of any deficiencies and providing information onhow to improve the trainees performance including showing the traineesperformance overlaid on an experts performance for the same task orsuggesting way to improve performance, providing information to thetrainee to improve performance deficiencies, and repeating the deficienttasks or the entire routine until proficient at each task and the entireroutine to a given level of competency, providing feedback to thetrainee/supervisor before, during, and/or after the trainee repeatsroutine/tasks. The methods also include processing and storing thetrainee repeat performance data in the databases, analyzing performancedata and modifying one, some, or all aspects of the training routine,task, avatar, hot spot, and/or environment features, properties,characteristics, and/or attributes to improve the routine, tasks, hotspots, and/or environment, analyzing performance data (expert andtrainee) and modifying one, some, and/or all aspects of the routine, thetask, the environment, the avatar, and/or the hot spot features,properties, characteristics, and/or attributes to improve any and allaspects of the training routine, the tasks, the environment, theavatar(s), and/or the hot spots to aid the trainee in those deficienttasks and allowing the trainee to redo the training task or the entiretraining session, and continuing the analyzing, determining, andimproving the routine, environment, avatar, hot spots, and/or hot spotcontent, wherein the improving may include revising, adding, deleting,modifying, and/or changing any some or all aspects of the trainingroutine, environment, avatar, hot spots, and/or hot spot content toenhance its effectiveness over time.

In certain embodiments, the methods further comprises generating anavatar assistant, wherein the avatar is configured to change formdepending on the task, routine, or program, may be generated using userinput, may be generated based on past trainee performance, or may adapta specific form to improve trainer environment and routine interaction.In other embodiments, the data includes data on how the expertaccomplished each training task, data on what hot spots the traineeactivates, data on the duration of time the expert spends in teach hotspot and data on the type of information viewed, data indicating a levelof competency in each task, data indicating an overall competency, anddata associated with ease of using each task, ease of using the entiretraining routine, and each of using and interacting with theenvironment.

Embodiments of this disclosure broadly relate to methods comprisingloading a pre-existing non VR or AR/MR/XR training task, routine, orprogram, generating information about (a) one, some, or all aspects,features, attributes, properties, and/or characteristics associated withthe training task, routine, or program and/or (b) one, some, or allaspects, features, attributes, properties, and/or characteristicsassociated equipment and/or devices used in the training task, routine,or program, gathering information about (a) one, some, or all of theaspects, features, attributes, properties, and/or characteristicsassociated with the training task, routine, or program and/or (b) one,some, or all of the aspects, features, attributes, properties, and/orcharacteristics associated the equipment and/or the devices used in thetraining task, routine, or program, and storing the generated and/orgathered information in one or more local databases and/or one or moreremote databases. The methods also include analyzing the pre-existingtraining program, the generated information, and the gatheredinformation, developing VR content, AR/MR/XR content, and/or VR andAR/MR/XR content for the pre-existing training program, generating a VRenvironment, an AR/MR/XR environment, or a VR and an AR/MR/XRenvironment for the training program, generating hot spots for (a) one,some, or all of the aspects, features, attributes, properties, and/orcharacteristics associated with the training task, routine, or programand/or (b) one, some, or all of the aspects, features, attributes,properties, and/or characteristics associated the equipment and/or thedevices used in the training task, routine, or program, and populatingthe environment with the generated hot spots. The methods also includesstarting an expert training session, collecting/capturing expert/trainerperformance and interaction data as an expert/trainer performs thetraining task, routine, or program, processing the expert/trainerperformance and interaction data, storing the expert/trainer performanceand interaction data in the databases, analyzing the collected/capturedperformance and interaction data, modifying one, some, or all aspects,features, attributes, properties, and/or characteristics of the trainingtask, routine, or, program, the environment, and/or hot spots, during orafter performing the task, routine, or program or any part thereof,updating the task, routine, or program, the environment and/or the hotspots, and storing the updated task, routine, or program in thedatabases. The methods also include providing feedback to the expertwith overall performance and/or part performance and interaction data togain insight from the expert on how to improve the training task,routine, or program, the environment, and/or the hot spots, conditionaltesting to permit the expert to repeat the entire task, routine, orprogram or any portion of the task, routine, or program, if the expertis not satisfied with any aspect of the trainer performance, thenrepeating the steps starting with the collecting/capturing step so thatthe expert repeats the entire task, routine, or program or any portionof the task, routine, or program. The methods also include if the expertis satisfied with the trainer performance, starting a trainee trainingsession, collecting/capturing trainee performance and interaction dataas a trainee performs the training task, routine, or program, processingthe trainee performance and interaction data, storing the traineeperformance and interaction data in the databases, analyzing thecollected/captured trainee performance and interaction data, modifyingone, some, or all aspects, features, attributes, properties, and/orcharacteristics of task, routine, or program, the environment, and/orhot spots, and updating the program and storing the updated program thedatabases, The methods also include providing feedback to anexpert/supervisor before, during, and/or after the trainee performance,providing feedback to the trainee including deficient task performancedata, conditional testing to permit the trainee to repeat the repeat theentire task, routine, or program or any portion of the task, routine, orprogram, if the trainee did not pass the training task, routine, orprogram or any part thereof, then repeating the steps starting with thecollecting/capturing step so that the expert repeats the entire task,routine, or program or any portion of the task, routine, or program, ifthe trainee passes the training task, routine, or program or the partthereof, processing the trainee pass performance and interaction data,storing the trainee pass performance and interaction data in thedatabases, analyzing the trainee pass performance and interaction data,modifying one, some, or all aspects of the training task, routine, orprogram, the environment and/or the hot spots, updating the program andstoring the updated training task, routine, or program in the databases,continuously, periodically, or intermittently analyzing all of thestored data, developing predictive tools or routines to assist traineesin performing the training task, routine, or program and/or navigatingthe environment, developing trainee type predictive tools to assistspecific types of trainees in performing the training program and/ornavigating the environment, and updating the task, routine, or program,the environment, the hot spots.

In certain embodiments, the training task, routine, or program includes:one or more devices, one or more device explanatory routines, one ormore device training routines, one or more device training tasks, one ormore question/answer sessions, one or more information presentationroutines, one or more trainee pass/fail routines, one or moreperformance ranking routines, one or more trainee feedback routines, orany combination thereof. In other embodiments, the VR content and/orAR/MR/XR content includes purely textual content, purely non-textualcontent, or mixtures of textual content and non-textual content. Inother embodiments, the methods further comprise analyzing the textualinformation and preparing CG content constructs based on the textualcontent, and analyzing the non-textual information and preparing CGcontent constructs based on the non-textual content. In otherembodiments, the CG content constructs comprise CG 2D, 3D, 4D, or nDconstructs. In other embodiments, the environment is fully interactivebased primarily on motion-based processing, hard select processing,gesture processing, or any combination thereof. In other embodiments,the hot spots comprise (a) specific hot spots associated specificaspects, features, attributes, properties, and/or characteristics of thetraining task, routine, or program or the environment, or (b) generalinformation hot spots associated with global aspects, features,attributes, properties, and/or characteristics of the training task,routine, or program or the environment. In other embodiments, theexpert/trainer or trainee performance and interaction data includes: (j)data on how the expert accomplishes the entire program, (k) data on howthe expert accomplishes each routine and/or task of the program, (1)data on what hot spots the expert activates, (m) data on the duration oftime the expert spends in each activated hot spot, (n) data on the typeof information viewed, (o) data indicating a level of competency inentire program and each routing and/or task, (p) data associated withease of performing each of the routine and/or task, (q) data associatedwith the ease/difficulty of using the entire training program, (r) dataassociated with the ease/difficulty of using each routine or task, (s)data associated with using and interacting with the entire environment,(t) data associated with using and interacting with one, some, or allaspects, features, attributes, properties, and/or characteristics of theprogram, environment, routines, tasks, content, and/or hot spots, (u)data that may be used to construct predictive tools or routines fortailoring the training program to individual trainees, (v) data on howthe trainee accomplishes the entire program, (w) data on how the traineeaccomplishes each task of the program, (x) data on what hot spots thetrainee activates, (y) data on the duration of time the trainee spendsin each activated hot spot, (z) data on the type of information viewed,data indicating a level of competency in entire routing and each task,(aa) data associated with ease of performing each of the routines ortasks, ease/difficulty of using the entire training program andease/difficulty of using each routine or task thereof, (bb) dataassociated with using and interacting with the entire environment and/orone, some, or all aspects, features, attributes, properties, and/orcharacteristics of the environment, routines, tasks, content, and/or hotspots, and (cc) mixtures or combinations thereof. In other embodiments,the processing comprises formatting the information for more efficientdatabase storage and retrieval. In other embodiments, the methodsfurther comprise generating one or more avatars, and providing theavatar with the expert performance and interaction data, and wherein inany of the analyzing, processing, modifying, storing, and/or updatingsteps, analyzing, processing, modifying, storing, and/or updating theone or more avatars.

Embodiments of this disclosure broadly relate to apparatuses, systems,or interfaces capable of performing any of the preceding claimscomprising at least one processing unit, at least one motion sensor, atleast one display device, at least one user feedback device, and atleast one user input device, wherein the at least on processing unit isin communication with the other devices.

Embodiments of this disclosure also broadly relate to methods includingdisplaying, via a processing unit, a virtual reality (VR), augmentedreality (AR), a mixed reality (MR), or an extended reality (XR)environment including a plurality of objects, a plurality of scenes,and/or a plurality of attributes associated with the environment and/orobjects or scenes on a display device in communication with at least oneprocessing unit; viewing, via the at least one processing unit, theenvironment through wearable display device such as an eye trackingdevice, a head up display, smart phone, tablet, or any other wearable ormobile or non-mobile viewing device in communication with the processingunit; superimposing, in or on the environment via the processing unit,an interactive control construct including a plurality of active controlareas; optionally displaying, via the processing unit, a selectionobject or indication for visualizing eye movement from the deviceglasses in the environment or in or on the control construct; sensingeye movement via the device glasses and moving the selection object orindication correspondingly; and if the movement is on or in a particularactive control area of the construct, then converting the movement intoa construct function associated with the particular active control areaof the construct; or if the movement is on or in the environment, thenconverting the movement into a selection and/or activation function of aparticular object or a selection, activation, scrolling and/or attributefunction of the environment.

In certain embodiments, the methods further comprise repeating thesensing and if steps, until an exit or stop command is selected. Inother embodiments, the methods further comprise prior to the environmentdisplaying step, sensing a first eye movement, wherein the firstmovement includes at least one property exceeding a threshold value orconfigured position. In other embodiments, the control areas comprise afast forward movement area, a no movement area, a backward movementarea, a pan left area, a pan right area, and a no pan area, wherein eacharea has a corresponding movement function, control function or acorresponding panning function associated therewith. In otherembodiments, the control areas comprise a plurality of bands, each ofthe bands having a plurality of regions. The bands may also includegradients of functions, attributes, groupings, or perspectives controls.the bands comprise a fast forward band, a slow forward band, a stop orno motion-band, a fast backward band, and a slow forward band, whereineach band has a corresponding movement function associated therewith,and the regions comprise a pan left region, a pan right region and a nopan region interposed therebetween, wherein each band has acorresponding panning/control function associated therewith. In otherembodiments, the methods further comprise sensing eye movement via thedevice and moving the selection object or indicator, visible orinvisible, correspondingly; if the movement is on or in a particularband and a particular region of the particular band, then converting themovement into the corresponding movement function and panning functionand moving through the environment accordingly; or if the movement is onor in the environment, then converting the movement into a selection,control, and/or activation function of a particular object or aselection, activation, and/or attribute function of the environment. Inother embodiments, the methods further comprise prior to the if steps,confirming the selection of the particular area of the construct, orconfirming the selection of the object or attribute in the environment.

Embodiments of this disclosure also broadly relate to apparatuses suchas a set of glasses in communication with a mobile or non-mobile device(e.g., smart phones, smart TV, tablets, laptop computers, desktopcomputer, etc. where all the processing is done on a connected device,including a display device; an input device; eye tracking device; and aprocessor or output from a processor, in communication with the displaydevice, the glasses and the input device, configured to: display avirtual reality (VR), augmented reality (AR), or a mixed reality (MR)environment including a plurality of objects, a plurality of scenes,and/or a plurality of attributes associated with the environment,objects and/or scenes on the display device; initiate the glasses toview the environment; superimpose, in or on the environment, aninteractive control construct including a plurality active control areasviewable through the glasses; display a selection object for visualizingeye movement from the glasses in or on the environment or in or on thecontrol construct; sense eye movement via the glasses and moving theselection object correspondingly; and if the movement is on or in aparticular active control area of the construct, then converting themovement into a function associated with the particular active controlarea of the construct; or if the movement is on or in the environment,then converting the movement into a selection and/or activation functionof a particular object or a selection, activation, and/or attributeadjustment function of a particular attribute of the particular objector of the environment.

In certain embodiments, the processor is further configured to: prior tothe environment display, sense a first movement from the glasses or fromthe input device, wherein the first movement includes at least onemovement property exceeding a threshold value or position. In otherembodiments, the device comprises a mobile device including the displaydevice, the input device, and the processing unit. In other embodiments,the device comprises a mobile device including the display device, theinput device, and the processing unit. In other embodiments, the displaydevice comprises a touchscreen and the input device is a motion sensor.In other embodiments, the mobile device comprises a cell phone, awearable device, a tablet computer, or a laptop computer.

Embodiments of this disclosure also broadly relate to methods includingdisplaying, via a processing unit, a plurality of icon on a displaydevice in communication with the processing unit;

sensing, via the processing unit, eye movement via eye tracking glassesin communication with the processing unit, where the movement isdiscernibly towards a particular icon; expanding, via the processingunit, the particular icon; activating, via the processing unit, theparticular icon once the movement enters the particular icon;displaying, via the processing unit, one or a plurality of attributeicons associated with the particular icon; sensing, via the processingunit, further eye movement via the glasses discernibly towards aparticular attribute icon; selecting and activating the particularattribute icon; and if the particular attribute icon is adjustable,sensing further eye movement via the glasses to adjust the attributevalue.

In certain embodiments, the methods further comprise repeating thesteps. In other embodiments, the methods further comprise prior to thedisplaying icon step, sensing a first eye movement, wherein the firstmovement includes at least one property exceeding a threshold value. Inother embodiments, the expanding step further includes displaying aconfirmation icon within the expanded particular icon; and theactivating step comprises: sensing eye movement via the glasses towardsthe confirmation icon; and activating the particular icon when themovement stop for a microhold over the confirmation icon. In otherembodiments, the activating step further includes a microhold over theparticular icon. In other embodiments, the selecting and activating theparticular attribute icon step further includes a microhold over theparticular attribute icon. In other embodiments, the selecting andactivating the particular attribute icon step further includes:expanding the particular attribute icon and displaying a confirmationicon within an expanded particular icon; and the activating stepcomprises: sensing eye movement via the glasses towards the confirmationicon; selecting and activating the particular attribute icon; and if theparticular attribute icon is adjustable, sensing further eye movementvia the glasses to adjust the attribute value.

Embodiments of this disclosure also broadly relate to apparatusesincluding a display device;

an input device; eye tracking glasses; and a processor, in communicationwith the display device, the glasses and the input device, configuredto: display a plurality of icon on a display device in communicationwith the processing unit; sense eye movement via eye tracking glasses,where the movement is discernibly towards a particular icon; expand theparticular icon; activate the particular icon once the movement entersthe particular icon; display one or a plurality of attribute iconsassociated with the particular icon; sense further eye movement via theglasses discernibly towards a particular attribute icon; select andactivate the particular attribute icon; if the particular attribute iconis adjustable, sense further eye movement via the glasses to adjust theattribute value.

In certain embodiments, the processor is further configured to: repeatthe display, sense, expand, activate, display, select and activate andattribute adjust. In other embodiments, the processor is furtherconfigured to: prior to the display icons, sense a first eye movement,wherein the first movement includes at least one property exceeding athreshold value. In other embodiments, the expand further includesdisplay of a confirmation icon within the expanded particular icon; andthe activate comprises: sense eye movement via the glasses towards theconfirmation icon; and activate the particular icon when the movementstop for a microhold over the confirmation icon. In other embodiments,the activate further includes a microhold over the particular icon. Inother embodiments, the select and activate the particular attribute iconstep further includes a microhold over the particular attribute icon. Inother embodiments, the select and activate the particular attribute iconstep further includes: expand the particular attribute icon and displaya confirmation icon within an expanded particular icon; and the activatecomprises: sense eye movement via the glasses towards the confirmationicon; select and activate the particular attribute icon; and if theparticular attribute icon is adjustable, sense further eye movement viathe glasses to adjust the attribute value.

Embodiments of this disclosure also broadly relate to methods including(a) capturing, via a processing unit, trainee images or a trainee imagesequence and trainer images and/or a trainer image sequence from atleast one motion sensor corresponding to a specific task; (b)constructing, via the processing unit, constructs of the trainee imagesor the trainee image sequence and constructs of the trainer images orthe trainer image sequence on an image by image basis and on a featureby feature basis; (c) scaling, via the processing unit, the traineeand/or trainer constructs on an image by image and on a feature byfeature basis so that the constructs are spatially and temporallyconformed; (d) comparing, via the processing unit, the scaled constructson an image by image basis and on a feature by feature basis; (e)determining, via the processing unit, differences between the scaledconstructs on an image by image basis and on a feature by feature basis;(f) overlaying, via the processing unit, the scaled trainee constructonto the scaled trainer construct on an image by image basis; and (g)displaying, via the processing unit, the overlaid constructs on an imageby image basis on a display device in communication with the processingunit.

In certain embodiments, the methods further comprise (h) highlighting,via the processing unit, the differences on an image by image basis andon a feature by feature basis on the display device. In otherembodiments, the methods further comprise (i) repeating the steps(a)-(h) until a stop function is invoked to stop the training exerciseor the differences satisfy at least one completion criteria. In otherembodiments, the at least one completion criteria comprises conformityor substantially conformity between the trainee scaled construct and thetrainer scaled construct on an image by image basis or the at least onecompletion criteria comprise minimizing one, some or all of differencesas evidenced by changes in the differences upon repeating the exercisebeing less than a minimal value, where the minimal value is less than orequal to a 5%, less than or equal to a 2.5%, less than or equal to a 2%,or less than or equal to a 1%. In other embodiments, the start and stopfunctions may be activated by motion-based processing, gesture basedprocessing or a hard select processing.

Embodiments of this disclosure also broadly relate to apparatuses adisplay device; an image capture device; an input device; an interfaceis configured to: capture trainee images or a trainee image sequencefrom the image capture device and trainer images or a trainer imagesequence from the image capture device; receive input from the inputdevice to invoke a start function or a stop function; and a processor isconfigured to: construct a trainee image by image construct and atrainer image by image construct; scale the trainee image by imageconstruct and/or trainer image by image construct so that the constructsare conformed spatially and temporally; compare the scaled constructs onan image by image basis; determine differences between the scaledconstructs on an image by image basis; overlay the scaled traineeconstruct onto the scaled trainer construct on an image by image basis;and display the overlaid constructs on an image by image basis. Incertain embodiments, the processor is further configured to: highlightthe differences on an image by image basis and feature by feature basis.In other embodiments, the processor is further configured to: repeat theconstruct, scale, compare, determine, overlay and display events untilthe a stop function is invoked to stop the training exercise or thedifferences satisfy at least one completion criteria.

Suitable Components for Use in the Disclosure Motion Sensors

Suitable motion sensors include, without limitation, optical sensors,acoustic sensors, thermal sensors, optoacoustic sensors, wave formsensors, pixel differentiators, or any other sensor or combination ofsensors that are capable of sensing movement or changes in movement, ormixtures and combinations thereof. Suitable motion sensing apparatusinclude, without limitation, motion sensors of any form such as digitalcameras, optical scanners, optical roller ball devices, touch pads,inductive pads, capacitive pads, holographic devices, laser trackingdevices, thermal devices, electromagnetic field (EMF) sensors, wave formsensors, any other device capable of sensing motion, changes in EMF,changes in a wave form, or the like or arrays of such devices ormixtures or combinations thereof. The sensors may be digital, analog, ora combination of digital and analog. The motion sensors may be touchpads, touchless pads, touch sensors, touchless sensors, inductivesensors, capacitive sensors, optical sensors, acoustic sensors, thermalsensors, optoacoustic sensors, electromagnetic field (EMF) sensors,strain gauges, accelerometers, pulse or waveform sensor, any othersensor that senses movement or changes in movement, or mixtures andcombinations thereof. The sensors may be digital, analog, or acombination of digital and analog or any other type. For camera systems,the systems may sense motion within a zone, area, or volume in front ofthe lens or a plurality of lens. Optical sensors include any sensorusing electromagnetic waves to detect movement or motion within inactive zone. The optical sensors may operate in any region of theelectromagnetic spectrum including, without limitation, radio frequency(RF), microwave, near infrared (IR), IR, far IR, visible, ultra violet(UV), or mixtures and combinations thereof. Exemplary optical sensorsinclude, without limitation, camera systems, the systems may sensemotion within a zone, area or volume in front of the lens. Acousticsensor may operate over the entire sonic range which includes the humanaudio range, animal audio ranges, other ranges capable of being sensedby devices, or mixtures and combinations thereof. EMF sensors may beused and operate in any frequency range of the electromagnetic spectrumor any waveform or field sensing device that are capable of discerningmotion with a given electromagnetic field (EMF), any other field, orcombination thereof. Moreover, LCD screen(s), other screens and/ordisplays may be incorporated to identify which devices are chosen or thetemperature setting, etc. Moreover, the interface may project a virtualcontrol surface and sense motion within the projected image and invokeactions based on the sensed motion. The motion sensor associated withthe interfaces of this disclosure may also be acoustic motion sensorusing any acceptable region of the sound spectrum. A volume of a liquidor gas, where a user's body part or object under the control of a usermay be immersed, may be used, where sensors associated with the liquidor gas can discern motion. Any sensor being able to discern differencesin transverse, longitudinal, pulse, compression or any other waveformmay be used to discern motion and any sensor measuring gravitational,magnetic, electro-magnetic, or electrical changes relating to motion orcontact while moving (resistive and capacitive screens) could be used.Of course, the interfaces can include mixtures or combinations of anyknown or yet to be invented motion sensors. The motion sensors may beused in conjunction with displays, keyboards, touch pads, touchlesspads, sensors of any type, or other devices associated with a computer,a notebook computer or a drawing tablet or any mobile or stationarydevice, and/or device, head worn device, or stationary device.

Suitable motion sensing apparatus include, without limitation, motionsensors of any form such as digital cameras, optical scanners, opticalroller ball devices, touch pads, inductive pads, capacitive pads,holographic devices, laser tracking devices, thermal devices, EMFsensors, wave form sensors, MEMS sensors, any other device capable ofsensing motion, changes in EMF, changes in wave form, or the like orarrays of such devices or mixtures or combinations thereof. Other motionsensors that sense changes in pressure, in stress and strain (straingauges), changes in surface coverage measured by sensors that measuresurface area or changes in surface are coverage, change in accelerationmeasured by accelerometers, or any other sensor that measures changes inforce, pressure, velocity, volume, gravity, acceleration, any otherforce sensor or mixtures and combinations thereof.

Controllable Objects

Suitable physical mechanical, electro-mechanical, magnetic,electro-magnetic, electrical, or electronic devices, hardware devices,appliances, biometric devices, automotive devices, VR objects, ARobjects, MR objects, and/or any other real world device and/or virtualobject that may be controlled by a processing unit include, withoutlimitation, any electrical and/or hardware device or appliance or VRobject that may or may not have attributes, all of which may becontrolled by a switch, a joy stick, a stick controller, other similartype controller, and/or software programs or objects. Exemplary examplesof such attributes include, without limitation, ON, OFF, intensityand/or amplitude, impedance, capacitance, inductance, softwareattributes, lists, submenus, layers, sublayers, other leveling formatsassociated with software programs, objects, haptics, any othercontrollable electrical and/or electro-mechanical function and/orattribute of the device and/or mixtures or combinations thereof.Exemplary examples of devices include, without limitation, environmentalcontrols, building systems and controls, lighting devices such as indoorand/or outdoor lights or light fixtures, cameras, ovens (conventional,convection, microwave, and/or etc.), dishwashers, stoves, sound systems,mobile devices, display systems (TVs, VCRs, DVDs, cable boxes, satelliteboxes, and/or etc.), alarm systems, control systems, air conditioningsystems (air conditions and heaters), energy management systems, medicaldevices, vehicles, robots, robotic control systems, UAVs, equipment andmachinery control systems, hot and cold water supply devices, airconditioning system, heating systems, fuel delivery systems, energymanagement systems, product delivery systems, ventilation systems, airhandling systems, computers and computer systems, chemical plant controlsystems, manufacturing plant control systems, computer operating systemsand other software systems, programs, routines, objects, and/orelements, remote control systems, or the like virtual and augmentedreality systems, holograms, and/or mixtures or combinations thereof.

Software Systems

Suitable software systems, software products, and/or software objectsthat are amenable to control by the interface of this disclosureinclude, without limitation, any analog or digital processing unit orunits having single or a plurality of software products installedthereon and where each software product has one or more adjustableattributes associated therewith, or singular software programs orsystems with one or more adjustable attributes, menus, lists, or otherfunctions, attributes, and/or characteristics, and/or display outputs.Exemplary examples of such software products include, withoutlimitation, operating systems, graphics systems, business softwaresystems, word processor systems, business systems, online merchandising,online merchandising systems, purchasing and business transactionsystems, databases, software programs and applications, internetbrowsers, accounting systems, military systems, control systems, VR, AR,MR systems or the like, or mixtures or combinations thereof. Softwareobjects generally refer to all components within a software system orproduct that are controllable by at least one processing unit.

Processing Units

Suitable processing units for use in the present disclosure include,without limitation, digital processing units (DPUs), analog processingunits (APUs), Field Programmable Gate Arrays (FPGAs), any othertechnology that may receive motion sensor output and generate commandand/or control functions for objects under the control of the processingunit, and/or mixtures and combinations thereof.

Suitable digital processing units (DPUs) include, without limitation,any digital processing unit capable of accepting input from a pluralityof devices and converting at least some of the input into outputdesigned to select and/or control attributes of one or more of thedevices. Exemplary examples of such DPUs include, without limitation,microprocessor, microcontrollers, or the like manufactured by Intel,Motorola, Ericsson, HP, Samsung, Hitachi, NRC, Applied Materials, AMD,Cyrix, Sun Microsystem, Philips, National Semiconductor, Qualcomm, orany other manufacture of microprocessors or microcontrollers, and/ormixtures or combinations thereof.

Suitable analog processing units (APUs) include, without limitation, anyanalog processing unit capable of accepting input from a plurality ofdevices and converting at least some of the input into output designedto control attributes of one or more of the devices. Such analog devicesare available from manufacturers such as Analog Devices Inc.

User Feedback Units

Suitable user feedback units include, without limitation, cathode raytubes, liquid crystal displays, light emitting diode displays, organiclight emitting diode displays, plasma displays, touch screens, touchsensitive input/output devices, audio input/output devices, audio-visualinput/output devices, holographic displays and environments, keyboardinput devices, mouse input devices, optical input devices, and any otherinput and/or output device that permits a user to receive user intendedinputs and generated output signals, and/or create input signals.

Predictive Training Methodology

The inventors have found that predictive virtual training systems,apparatuses, interfaces, and methods for implementing them may beconstructed including one or more processing units, one or more motionsensing devices or motion sensors, optionally one or more non-motionsensors, one or more input devices, and one or more output devices suchas one or more display devices, wherein the processing unit includes avirtual training program and is configured to (a) output the trainingprogram in response to user input data sensed by the sensors or receivedfrom the input devices, (b) collect user interaction data whileperforming the virtual training program, and (c) modify, alter, change,augment, update, enhance, reformat, restructure, and/or redesign thevirtual training program to better tailor the virtual training programfor each user, for each user, type and/or for all users.

DETAILED DESCRIPTION OF THE DRAWINGS OF THE DISCLOSURE Virtual TrainingConceptual Flow Chart

Referring now to FIG. 1A, an embodiment of a schematic flow chart of amethod of training, generally 100, is shown to include a start step 102.After starting, the method includes a trainer capturing step 104, wheretrainer images or a trainer image sequence corresponding to a trainerperforming a training exercise or routine or program including, withoutlimitation, a game such as golf, baseball, football, soccer, etc., anexercise routine, a martial art routine, a dance routine, an instrumentroutine, or any other training routine or program that may be capturedby an imaging input device. The method also includes a trainerconstructing step 106, where computer generated trainer constructs on animage by image basis. The method also includes a trainee capturing step108, where trainee images or a trainee image sequence corresponding tothe trainee performing the training exercise or routine or program. Themethod also includes a trainee constructing step 110, where computergenerated trainee constructs on an image by image basis.

In certain embodiments, the construct comprise a polygon (triangles andhigher polygons) construct, where all features of the images arerepresented by a plurality of polygons capable of being scaled so thatthe trainer image features may be conformed to the trainer imagefeatures for comparison and overlaying purposes. In other embodiments,the construct comprise a tessellated construct, again capable of beingscaled so that the trainer image features may be conformed to thetrainer image features for comparison and overlaying purposes. In otherembodiments, the construct is a finite element construct, again capableof being scaled so that the trainer image features may be conformed tothe trainer image features for comparison and overlaying purposes.

The method also includes a scaling step 112, where the trainee and/ortrainer constructs are scaled to correspond to each other on an image byimage basis. The scaling adjusts corresponding features in the traineeimages to corresponding features in the trainer images. For example, thetrainee body is scaled to conform to the trainer body so that themovement of the trainee and trainer may be compared. As the images arecaptured over time, the scaling may include both spatial and temporalscaling. The method also includes a comparing step 114, where the scaledtrainee and trainer constructs are compared on an image by image basisand where comparison is on a feature by feature basis as well. Themethod also includes a determining step 116, where differences betweenthe features on an image by image basis are generated. The method alsoincludes a overlaying step 118, where the scaled trainee construct ontothe scaled trainer construct on an image by image basis. The method alsoincludes a displaying step 120, where the overlaid scaled constructs aredisplayed on a display device to show the difference between the traineeperforming the training exercise/routine/program and the trainerperforming the training exercise/routine/program.

The method also include a differences test step 122, where thedifferences between the trainer performance on an image by image basisare analyzes to determine if the differences meet one or more minimumdifference criteria such as whole body position differences on an imageby image basis, body part position differences on an image by imagebasis, body position temporal difference on an image sequence basis,body part position temporal differences on an image sequence basis, anyother differences, or any combination thereof, where the image sequencemay be the entire number of captured images or image sequence or anypart thereof. Additionally, the image capturing may be on any time framefrom near continuous to images from a continuous capture based on agiven time increment such as between a 1 millisecond time increment anda 10 second time increment or any increment therebetween. If thedifference exceed the one or more minimum difference criteria, thencontrol is transferred along a NO path back to the capturing of thetrainee image step 108. If the difference is less than or equal to theone or more minimum difference criteria, then control is transferredalong a YES path to a stop step 124. While the method shows a specificsequence of steps, one or more of these steps may be performed in adifferent sequence or simultaneously or in any other combinationprovided that the end result is to provide a trainee information forcorrecting a trainee performance relative to a trainer performance.

Referring now to FIG. 1B, another embodiment of a schematic flow chartof a method of training, generally 100, including a start step 102.After starting, the method includes a trainer capturing step 104, wheretrainer images or a trainer image sequence corresponding to a trainerperforming a training exercise or routine or program including, withoutlimitation, a game such as golf, baseball, football, soccer, etc., anexercise routine, a martial art routine, a dance routine, an instrumentroutine, or any other training routine or program that may be capturedby an imaging input device. The method also includes a trainerconstructing step 106, where computer generated trainer constructs on animage by image basis. The method also includes a trainee capturing step108, where trainee images or a trainee image sequence corresponding tothe trainee performing the training exercise or routine or program. Themethod also includes a trainee constructing step 110, where computergenerated trainee constructs on an image by image basis.

In certain embodiments, the construct comprise a polygon (triangles andhigher polygons) construct, where all features of the images arerepresented by a plurality of polygons capable of being scaled so thatthe trainer image features may be conformed to the trainer imagefeatures for comparison and overlaying purposes. In other embodiments,the construct comprise a tessellated construct, again capable of beingscaled so that the trainer image features may be conformed to thetrainer image features for comparison and overlaying purposes. In otherembodiments, the construct is a finite element construct, again capableof being scaled so that the trainer image features may be conformed tothe trainer image features for comparison and overlaying purposes.

The method also includes a scaling step 112, where the trainee and/ortrainer constructs are scaled to correspond to each other on an image byimage basis. The scaling adjusts corresponding features in the traineeimages to corresponding features in the trainer images. For example, thetrainee body is scaled to conform to the trainer body so that themovement of the trainee and trainer may be compared. As the images arecaptured over time, the scaling may include both spatial and temporalscaling. The method also includes a comparing step 114, where the scaledtrainee and trainer constructs are compared on an image by image basisand where comparison is on a feature by feature basis as well. Themethod also includes a determining step 116, where differences betweenthe features on an image by image basis are generated. The method alsoincludes a overlaying step 118, where the scaled trainee construct ontothe scaled trainer construct on an image by image basis. The method alsoincludes a displaying step 120, where the overlaid scaled constructs aredisplayed on a display device to show the difference between the traineeperforming the training exercise/routine/program and the trainerperforming the training exercise/routine/program. The method alsoincludes a highlighting step 126, where the differences are displayed onthe display device in a highlighted format (color, shade, pulsating, orother highlighting property) as the constructs are being displayed sothat the trainee and/or trainer may see the differences.

The method also include a differences test step 122, where thedifferences between the trainer performance on an image by image basisare analyzes to determine if the differences meet one or more minimumdifference criteria such as whole body position differences on an imageby image basis, body part position differences on an image by imagebasis, body position temporal difference on an image sequence basis,body part position temporal differences on an image sequence basis, anyother differences, or any combination thereof, where the image sequencemay be the entire number of captured images or image sequence or anypart thereof. Additionally, the image capturing may be on any time framefrom near continuous to images from a continuous capture based on agiven time increment such as between a 1 millisecond time increment anda 10 second time increment or any increment therebetween. If thedifference exceed the one or more minimum difference criteria, thencontrol is transferred along a NO path back to the capturing of thetrainee image step 108. If the difference is less than or equal to theone or more minimum difference criteria, then control is transferredalong a YES path to a stop step 124. While the method shows a specificsequence of steps, one or more of these steps may be performed in adifferent sequence or simultaneously or in any other combinationprovided that the end result is to provide a trainee information forcorrecting a trainee performance relative to a trainer performance.

Referring now to FIG. 1C, another embodiment of a schematic flow chartof a method of training, generally 100, including a start step 102.After starting, the method includes a trainer capturing step 104, wheretrainer images or a trainer image sequence corresponding to a trainerperforming a training exercise or routine or program including, withoutlimitation, a game such as golf, baseball, football, soccer, etc., anexercise routine, a martial art routine, a dance routine, an instrumentroutine, or any other training routine or program that may be capturedby an imaging input device. The method also includes a trainerconstructing step 106, where computer generated trainer constructs on animage by image basis. The method also includes a trainee capturing step108, where trainee images or a trainee image sequence corresponding tothe trainee performing the training exercise or routine or program. Themethod also includes a trainee constructing step 110, where computergenerated trainee constructs on an image by image basis.

In certain embodiments, the construct comprise a polygon (triangles andhigher polygons) construct, where all features of the images arerepresented by a plurality of polygons capable of being scaled so thatthe trainer image features may be conformed to the trainer imagefeatures for comparison and overlaying purposes. In other embodiments,the construct comprise a tessellated construct, again capable of beingscaled so that the trainer image features may be conformed to thetrainer image features for comparison and overlaying purposes. In otherembodiments, the construct is a finite element construct, again capableof being scaled so that the trainer image features may be conformed tothe trainer image features for comparison and overlaying purposes.

The method also includes a scaling step 112, where the trainee and/ortrainer constructs are scaled to correspond to each other on an image byimage basis. The scaling adjusts corresponding features in the traineeimages to corresponding features in the trainer images. For example, thetrainee body is scaled to conform to the trainer body so that themovement of the trainee and trainer may be compared. As the images arecaptured over time, the scaling may include both spatial and temporalscaling. The method also includes a comparing step 114, where the scaledtrainee and trainer constructs are compared on an image by image basisand where comparison is on a feature by feature basis as well. Themethod also includes a determining step 116, where differences betweenthe features on an image by image basis are generated. The method alsoincludes a overlaying step 118, where the scaled trainee construct ontothe scaled trainer construct on an image by image basis. The method alsoincludes a displaying step 120, where the overlaid scaled constructs aredisplayed on a display device to show the difference between the traineeperforming the training exercise/routine/program and the trainerperforming the training exercise/routine/program. The method alsoincludes a highlighting step 126, where the differences are displayed onthe display device in a highlighted format (color, shade, pulsating, orother highlighting property) as the constructs are being displayed sothat the trainee and/or trainer may see the differences. The method alsoincludes an illustrating step 128, where the apparatuses, systems,and/or interfaces are implemented so that they may illustrate how atrainee may adjust body and/or body part positioning in each image orover a sequence of images so that the trainee may see how to adjust tominimize the highlighted differences.

The method also include a differences test step 122, where thedifferences between the trainer performance on an image by image basisare analyzes to determine if the differences meet one or more minimumdifference criteria such as whole body position differences on an imageby image basis, body part position differences on an image by imagebasis, body position temporal difference on an image sequence basis,body part position temporal differences on an image sequence basis, anyother differences, or any combination thereof, where the image sequencemay be the entire number of captured images or image sequence or anypart thereof. Additionally, the image capturing may be on any time framefrom near continuous to images from a continuous capture based on agiven time increment such as between a 1 millisecond time increment anda 10 second time increment or any increment therebetween. If thedifference exceed the one or more minimum difference criteria, thencontrol is transferred along a NO path back to the capturing of thetrainee image step 108. If the difference is less than or equal to theone or more minimum difference criteria, then control is transferredalong a YES path to a stop step 124. While the method shows a specificsequence of steps, one or more of these steps may be performed in adifferent sequence or simultaneously or in any other combinationprovided that the end result is to provide a trainee information forcorrecting a trainee performance relative to a trainer performance.

In certain embodiments, the trainee performance steps are repeated untila stop function is invoked using an input device or the differences areless than or equal to the one or more difference criteria, which may bethe same or different from the image capture device or the differencessatisfy at least one completion criteria, where the criteria maycomprise conformity or substantially conformity between the traineescaled construct and the trainer scaled construct on an image by imagebasis and feature by feature basis or the at least one completioncriteria comprise minimizing one, some or all of differences asevidenced by changes in the differences upon repeating the exercisebeing less than a minimal value. In certain embodiments, the minimalvalues are less than or equal to a 5%, less than or equal to a 2.5%,less than or equal to a 2%, or less than or equal to a 1%. For example,if the training exercise is a golf swing, then the features may includethe club and the body and the differences may include the club and handplacement, swing propagation, club angle changes, hand, arm, shoulder,back, hip and leg motion, and/or follow through. All such differencesmay be highlighted during the play back. By repeating the exercise, thetrainee may correct the highlighted differences until the differencesare below or completion criteria or minimized.

In these embodiments, the trainee may repeat the entire routine or anypart of the routine and in fact the method may suggest which part of theroutine to focus on and repeat. The method of repeating the traineeperformance may be customized to each trainee based on the traineeproclivities and learning process visual, auditory, etc.

Embodiment Interacting with nD Environments Using Multiple User FeedbackDevices

Referring now to FIG. 2, another embodiment of systems, apparatuses,and/or interfaces using two user feedback devices, generally as 200, isshown to include a 3D environment 202 displayed on a display device 204.The 3D environment 202 include a plurality of objects distributedthroughout the 3D environment. Here, the objects include a wall typeobject 206, a large cylindrical type object 208, a small cylindricaltype object 210, a large box type object 212, a small box type object214, a table chair 216, a table 218, a tea pot 220, and a chair 222. Thedisplay device 204 also includes a display cursor 224. The entire 3Denvironment 202 is viewable through a pair of eye/head tracking glasses(not shown). The two user feedback devices 200 may be a standard displaydevices and an eye/head tracking device capable of displaying an image.

When viewed through the eye/head tracking device, a virtual interactivecontroller 250 appears superimposed on the environment 202 and aneye/heading tracking cursor 252 also appears superimposed on theenvironment 202. The controller 250 also include a motion control area270 and a 3D environment object control area 290. The area 270 includefive bands 272 a-e. The bands 272 a-e are divided into three zones 274a-c resulting in 15 subzones labeled from upper left to lower right withthe following designations: FwF-L, FwS-L, L, BwS-L, BwF-L, FwF, FwS, DZ,BwS, BwF, FwF-R, FwS-R, L, BwS-R, and BwF-R. Fw means forward. Bw meansbackward. L means left and R means right.

The subzones FwF-L, FwS-L, L, BwS-L, BwF-L, FwF-R, FwS-R, L, BwS-R, andBwF-R are all relative zones. If the subzones FwF-L, FwS-L, L, BwS-L,BwF-L, FwF-R, FwS-R, L, BwS-R, and BwF-R are panning zones, then thecloser the cursors 252 is to the left edge of the FwF-L, FwS-L, L,BwS-L, and BwF-L subzones or right edge of the FwF-R, FwS-R, L, BwS-R,and BwF-R subzones, the faster the systems pan left or right through theenvironment 202, while the closer to the zones FwF, FwS, DZ, BwS, orBwF, the slower the systems pans left or right through the environment202. It should be recognized that positioning the cursors 224 and/or 252within one of these subzones causes the systems to move left or rightand simultaneously forward or backward at the indicated rate, a faterate, a moderate rate, or a slow rate.

Alternatively, if the subzones are rotational subzones, then the closerto banner left edge or banner right edge, the faster the environment 202rotates about the location of the cursors 224 and/or 252, while thecloser to the FwF, FwS, DZ, BwS, and BwF subzones, the slower therotation of the environment 202 about the location.

The FwF, FwS, DZ, BwS, or BwF subzones are not relative zones, but arezones that move forward or backward within the environment 202. The DZsubzone is a dead zone, meaning that locating the cursors 264 and/or 252in this subzone causes the systems to stop all movement within theenvironment 202. Positioning the cursors 224 and/or 252 within the FwFsubzone causes the systems to move the forward through the environment202 at a fast rate, while positioning the cursors 224 and/or 252 withinthe BwF subzone causes the systems to move the backward through theenvironment 202 at a fast rate. Positioning the cursors 224 and/or 252within the FwS subzone causes the systems to move the forward throughthe environment 202 at a slow rate, while positioning the cursors 224and/or 252 within the BwS subzone causes the systems to move thebackward through the environment 202 at a slow rate. Positioning thecursors 224 and/or 252 on the boundary of the FwF subzone and the FwSsubzone causes the systems to move the forward through the environment202 at a moderate rate, while positioning the cursors 224 and/or 252 onthe boundary of the BwF subzone and the BwS subzone causes the systemsto move the forward through the environment 202 at a moderate rate.

In other embodiments, the bands 272 a-e may represent zoom in and zoomout controls instead of moving forwards or backwards within the 3Denvironment 202 without changing the aspect ratio of the view.

The 3D environment object control area 290 includes objects a through i,which corresponds to the objects in the 3D environment 202. Thus, theobject a corresponds to the wall 206; the object b corresponds to thelarge cylinder 208; the object c corresponds to the small cylinder 210;the object d corresponds to the large box 212; the object e correspondsto the small box 214; the object f corresponds to the table chair 216;the object g corresponds to the table 218; the object h corresponds tothe tea pot 220; and the object i corresponds to the chair 208. Bymoving the cursors 224 and/or 252 into the area 290, the systems mayinvoke a scroll function that scrolls through the object, with asubsequent movement of the cursors 224 and/or 252 selects a particularobject. Once selected, the attributes associated with the corresponding3D environment object may be selected and changed by motion as set forthherein.

Embodiment Interacting with nD Environments Using Multiple User FeedbackDevices

Referring now to FIG. 3A, another embodiment of systems, apparatuses,and/or interfaces using two user feedback devices, generally 300, isshown to include a 3D environment 302 displayed on a display device 304including a display cursor 306. The 3D environment 302 include aplurality of objects a through jj distributed throughout the 3Denvironment 302. The objects a-jj are represented here by spheres forconvenience, but may actually be any type of object. The entire 3Denvironment 302 is viewable through a pair of eye/head tracking glasses(not shown). The two user feedback devices 200 may be a standard displaydevices and an eye/head tracking device capable of displaying an image.

When viewed through the eye/head tracking glasses, a virtual motioncontrol area 350, a virtual object control area 370, and an eye/headingtracking cursor 352 all appear superimposed on the environment 302. Thearea 350 includes nine zones 354 a-i, while the area 370 includesselectable objects a-jj, which correspond to the 3D environment objectsa-jj.

The zone 354 a is a dead zone so that when the cursor 306 and/or 352 ismoved into the zone 354 a, the systems cause all movement within the 3Denvironment 302 to stop.

The zones 354 b-e are pure relative motion zones meaning thatpositioning the cursor 306 and/or 352 within one of these zones causesthe systems to move through the environment 302 in the indicateddirection. For example, positioning the cursors 306 and/or 352 in thezone 354 b causes the systems to move left at a rate determined by howclose the cursors 306 and/or 352 is to the left edge of the banner 350.If the cursors 306 and/or 352 is close to the left edge, then thesystems move through the environment 302 at a fast rate; while if thecursors 306 and/or 352 is close to the dead zone 354 a, then the rate isslow. Positioning the cursors 306 and/or 352 in any of the other purerelative motion zones 354 c-e causes the systems to move through theenvironment 302 similarly.

The relative motion zones 354 f-i are simultaneous or compound relativemotion zones meaning that positioning the cursors 306 and/or 352 causesthe systems to move through the environment 302 in the indicateddirections. For example, positioning the cursors 306 and/or 352 in thezone 354 f causes the systems to move left and forward through theenvironment 302 at a rate determined by how close the cursors 306 and/or352 is to the top/left corner. If the cursors 306 or 352 is close totop/left corner, then the systems move through the environment 302 tothe left and forward at a fast rate; while if the cursors 306 and/or 352is close to the dead zone 354 a (bottom/right corner), then theleft/forward motion rate is slow. Positioning the cursors 306 and/or 352in any of the other compound relative motion zones 354 g-i causes thesystems to move through the environment 302 similarly.

The area 370 is a selection area so that when the cursors 306 and/or 352is positioned within the area 370. Once inside the area 370, movementtoward any direction will permit the user to select from the objects. Ifthe direction is aligned with more than one selectable object, then thesystems will causes those object to be highlighted or draw to or pushedaway from the cursors 306 or 352 so that further movement will result indiscrimination between the possible selectable objects as set forthherein.

Referring now to FIG. 3B, another embodiment of systems, apparatuses,and/or interfaces using two user feedback devices, generally 300, isshown to include a 3D environment 302 displayed on a display device 304including a display cursor 306. The 3D environment 302 include aplurality of objects a through jj distributed throughout the 3Denvironment 302. The objects a-jj are represented here by spheres forconvenience, but may actually be any type of object. The entire 3Denvironment 302 is viewable through a pair of eye/head tracking glasses(not shown).

When viewed through the eye/head tracking glasses, a virtual motioncontrol area 350, a virtual object control area 370, and an eye/headingtracking cursor 352 all appear superimposed on the environment 302. Thearea 350 includes nine zones 354 a-i, where the zones 354 b-i are all 3Drelative control zones meaning that positioning the cursor 306 or 352within any of these zones causes the systems to move in two or threedirection simultaneously. The area 370 includes selectable objects a-jj,which correspond to the 3D environment objects a-jj.

The zone 354 a is a dead zone so that when the cursors 306 and/or 352 ismoved into the zone 354 a, the systems cause all movement within the 3Denvironment 302 to stop.

The zones 354 b-e are two directional relative motion zones meaning thatpositioning the cursors 306 and/or 352 within one of these zones causesthe systems to move up or down and left or right or forward or backwardthrough the environment 302. For example, positioning the cursors 306 or352 in the zone 354 b causes the systems to move left and up or down ata rate determined by how close the cursors 306 and/or 352 is to the leftedge, top edge, bottom edge, right edge of the zone 354 b. If thecursors 306 and/or 352 is close to the left/top edge, then the systemsmove through the environment 302 at a fast rate up and to the left moveup pan left; while if the cursors 306 and/or 352 is close to theright/bottom edge or near the dead zone 354 a, then the rate is up andleft is slow move up and pan left at a slow rate. Positioning thecursors 302 and/or 352 in any of the other two directional relativemotion zones 354 c-e causes the systems to move through the environment302 similarly.

The zones 354 f-i are three directional relative motion zones meaningthat positioning the cursors 306 or 352 causes the systems to movethrough the environment 302 in the three directions simultaneously. Forexample, positioning the cursors 306 and/or 352 in the zone 354 f causesthe systems to move left, forward, and up or down through theenvironment 302 at a rate determined by how close the cursors 306 and/or352 is to the one of the edges or corners. If the cursors 306 and/or 352is close to top/left corner, then the systems move through theenvironment 302 to the left, forward, and up at a fast rate; while ifthe cursors 306 and/or 352 is close to the bottom/right corner (i.e.,close to the dead zone 354 a), then the systems move through theenvironment 302 to the left, forward, and down at a slow rate.Positioning the eye/heading tracking cursors 306 or 352 in any of theother compound relative motion zones 354 g-i causes the systems to movethrough the environment 302 similarly.

The area 370 is a selection area so that when the cursors 306 and/or 352is positioned within the area 370, movement toward any direction willpermit the user to select from the objects. If the direction is alignedwith more than one selectable objects, then the systems will cause thoseobject to be highlighted or draw to or pushed away from the cursors 306and/or 352 so that further movement will result in discriminationbetween the possible selectable objects as set forth herein.

Embodiment Interacting with nD Environments Using Multiple User FeedbackDevices

Referring now to FIG. 4, another embodiment of systems, apparatuses,and/or interfaces using two user feedback devices, generally 400, isshown to include a 3D environment 402 displayed on a display device 404including a display cursor 406. The 3D environment 402 include aplurality of objects a through jj distributed throughout the 3Denvironment 402. The objects a-jj are represented here by spheres forconvenience, but may actually be any type of object. The entire 3Denvironment 402 is viewable through a pair of eye/head tracking glasses(not shown).

When viewed through the eye/head tracking glasses, a virtual motioncontrol area 450, a virtual object control area 470, and an eye/headingtracking cursor 452 all appear superimposed on the environment 402. Thearea 450 includes nine zones 454 a-i, while the area 470 includesselectable objects a-jj, which correspond to the 3D environment objectsa-jj and are arranged in a flat projection onto the area 470 in theirprojected positions.

The zone 454 a is a dead zone so that when the cursors 406 and/or 452 ismoved into the zone 454 a, the systems cause all movement within the 3Denvironment 402 to stop.

The zones 454 b-e are pure relative motion zones meaning thatpositioning the cursors 406 and/or 452 within one of these zones causesthe systems to move through the environment 402 in the indicateddirection. For example, positioning the cursors 406 and/or 452 in thezone 454 b causes the systems to move left at a rate determined by howclose the cursor 406 or 452 is to the left edge of the banner 450. Ifthe cursors 406 and/or 452 is close to the left edge, then the systemsmove through the environment 402 at a fast rate; while if the cursors406 and/or 452 is close to the dead zone 454 a, then the rate is slow.Positioning the cursors 406 and/or 452 in any of the other pure relativemotion zones 454 c-e causes the systems to move through the environment402 similarly.

The relative motion zones 454 f-i are simultaneous or compound relativemotion zones meaning that positioning the cursors 406 and/or 452 causesthe systems to move through the environment 402 in the indicateddirections. For example, positioning the cursors 406 and/or 452 in thezone 454 f causes the systems to move left and forward through theenvironment 402 at a rate determined by how close the cursors 406 and/or452 is to the top/left corner. If the cursor 406 or 452 is close totop/left corner, then the systems move through the environment 402 tothe left and forward at a fast rate; while if the cursors 406 and/or 452is close to the dead zone 454 a (bottom/right corner), then theleft/forward motion rate is slow. Positioning the cursors 406 and/or 452in any of the other compound relative motion zones 454 g-i causes thesystems to move through the environment 402 similarly.

The area 470 is a selection area so that when the cursors 406 and/or 452is positioned within the area 470. Once inside the area 470, movementtoward any direction will permit the user to select from the objects. Ifthe direction is aligned with more than one selectable object, then thesystems will causes those object to be highlighted or draw to or pushedaway from the cursors 406 and/or 452 so that further movement willresult in discrimination between the possible selectable objects as setforth herein.

Embodiment Interacting with nD Environments Using Multiple User FeedbackDevices

Referring now to FIGS. 5A-D, another embodiment of systems, apparatuses,and/or interfaces using two user feedback devices, generally 500, isshown to include a 3D environment 502 displayed on a display device 504and cursors 503 and 505. The entire 3D environment 502 includes aplurality of levels 506 _(1-n), where n is an integer, as shown in FIG.5A. Each level 506 _(i) includes a plurality of sublevels 508_(1-m)where m is an integer, as shown in FIG. 5B. Each sublevel 508 _(j)includes a plurality of zones 510 _(1-q), where q is an integer, asshown in FIG. 5C. Each zone 510 _(k) includes three pluralities ofdifferent type of objects a-jj, 1-24 and α, β, γ, δ, ε, ζ, and η, asshown in FIG. 5D. Traversal through the levels, sublevels, and zones maybe affected using a display cursor 512 and/or a eye/head tracking cursor514. Traversal through the levels, sublevels, and zones may use anyselection methodology disclosed herein including motion only processingor using point and click methodology or other hard selectionmethodology. Additionally, the levels, sublevels, and/or zone may bepresented as a menu or list so that any methodology of this disclosuremay be used scrolling, selecting, activating, attribute adjusting, etc.Moreover, once a zone has been selected, the eye/head tracking glassesmay allow a virtual control area or areas to appear superimposed on theenvironment for control as set forth above.

Illustrative Virtual Training System Implementations

Geologic or Earth Teaching Embodiments

Referring now to FIGS. 6A-D, an embodiment of a system, apparatus,and/or interface of this disclosure, generally 600, illustrates ageological image and a set or plurality of hot spots or active areasused in a training routine.

Looking at FIGS. 6A&B, the system, apparatus, and/or interface is shownto include a display device 602 and an image area 604 containing a frontview of a geological image 606 and an aerial view of the geologicalimage 606. The system, apparatus, and/or interface analyzes the image606 and displays a set of hot spots or active areas a-g, represented bytransparent circular selectable objects. The hot spots a-g areassociated with certain relevant features of the geological image 606. Atrainee/student may activate each of the hot spots circled a-g using anyselecting procedure including motion, gestures, mouse selecting,touching, etc.

Looking at FIG. 6C, the display area 604 is split into an image displayarea 604 a and a content display area 604 b, when the trainee/studentselects hot spot d indicated by the hot spot d changing color or someother attribute of the hot spot d, which also appears highlighted in thecontent display area 604 b along with selectable content objects 1-10associated with the hot spot d that include relevant information aboutthe hot spot d. The selectable content objects 1-10 are shown heredistributed about a central selection object 608, which may or may notbe visible. Each one of the selectable objects 1-10 may includeinformation and/or tasks that the trainee/student must master concerningthe geological image 606. Of course, it should be recognized that theselectable content objects 1-10 may be distributed in any pattern withinthe display area 604 b, including a random or ordered pattern and thenumber of selectable content objects may be any number from 1 to 1000 ormore depending on the amount of information that may be associated witheach hot spot.

Looking at FIG. 6D, in an alternate embodiment, the system, apparatus,and/or interface changes the color of the hot spot d, when selected anddisplays the selectable content objects 1-10 associated with the hotspot d distributed above the hot spot d. Of course, the selectablecontent objects 1-10 may be distributed in any pattern within thedisplay area 604 b, including a random or ordered pattern. Moreover, thenon-selected hot spot a-c and e-g may disappears so that the selectablecontent objects 1-10 may surround the hot spot d. Selection of theselectable content objects 1-10 may be by any selection methodology.

The system, apparatus, and/or interface may also be configured togenerate a 3D environment based on the geological image 606. The system,apparatus, and/or interface also associated the hot spots a-g with theirassociates features of the geological image 606. The trainee/student maythen travel in the 3D environment changing viewpoints, looking atfeatures, activating any of the hot spots a-g and reviewing andperforming any tasks associated with the hot spots. A trainee maytravels anywhere within the 3D environment. As the trainee travelsthrough the environment, object may be expanded and hot spots associatedwith different aspects of expanded components will change along with thetype of information that is presented upon activating one of the hotspots. Hot spot content may also change based on the level of trainingand each hot spot may have training exercises associated with thefeature the hot spot is associated with. The hot spots in allembodiments may provide access into any content regardless of its sourceso that the hot spots may allow trainees to search the web or othersites for information on a particular aspect of the component with whichthe hot spot is associated. The web access may be configured so thatonly information on that particular feature is available to the trainee.The system, apparatus, and/or interface may therefore also include a webpre-interface that acts to restrict access except to the material neededby the trainee to complete the training or to complete a given taskassociated with the training routine.

Aircraft Cockpit Training Embodiments

Referring now to FIGS. 7A-D, an embodiment of a system, apparatus,and/or interface of this disclosure, generally 700, illustrates to animage of an aircraft cockpit image and a set or plurality of hot spotsor active areas used in a training routine.

Looking at FIG. 7A, the system, apparatus, and/or interface is shown toinclude a display device 702 and an image area 704 containing an insideview of an aircraft cockpit image 706. The system, apparatus, and/orinterface analyzes the image 706 and displays a set of hot spots oractive areas a-r, represented by transparent circular selectableobjects. The hot spots a-r are associated with certain relevant featuresof the geological image 706. A trainee/student may activate each of thehot spots circled a-r using any selecting procedure including motion,gestures, mouse selecting, touching, etc.

Looking at FIG. 7B, the display area 704 is split into an image displayarea 704 a and a content display area 704 b, when the trainee/studentselects hot spot m indicated by the hot spot m changing color or someother attribute of the hot spot m, which also appears highlighted in thecontent display area 704 b along with selectable content objects 1-10associated with the hot spot m that include relevant information aboutthe hot spot m. The selectable content objects 1-10 are shown heredistributed about a central selection object 708, which may or may notbe visible. Each one of the selectable objects 1-10 may includeinformation and/or tasks that the trainee/student must master concerningthe geological image 706. Of course, it should be recognized that theselectable content objects 1-10 may be distributed in any pattern withinthe display area 704 b, including a random or ordered pattern and thenumber of selectable content objects may be any number from 1 to 1000 ormore depending on the amount of information that may be associated witheach hot spot.

Looking at FIG. 7C, in an alternate embodiment, the system, apparatus,and/or interface changes the color of the hot spot m, when selected anddisplays the selectable content objects 1-10 associated with the hotspot m distributed above the hot spot m. Of course, the selectablecontent objects 1-10 may be distributed in any pattern within thedisplay area 704 b, including a random or ordered pattern. Moreover, thenon-selected hot spot a-l and n-r may disappears so that the selectablecontent objects 1-10 may surround the hot spot m. Selection of theselectable content objects 1-10 may be by any selection methodology.

Referring now to FIG. 7D, the system, apparatus, and/or interfacegenerates a 3D environment 710 based on the cockpit image 706. Thesystem, apparatus, and/or interface also associates the hot spots a-rwith their corresponding features in the 3D environment 710. Thetrainee/student may then travel in the 3D environment 710 changingviewpoints, looking at features, activating any of the hot spots a-r andreviewing associated content associated therewith and performing anytasks associated with the hot spots a-r. Again, a trainee may travelsanywhere within the 3D environment. As the trainee travels through theenvironment, object may be expanded and hot spots associated withdifferent aspects of expanded components will change along with the typeof information that is presented upon activating one of the hot spots.Hot spot content may also change based on the level of training and eachhot spot may have training exercises associated with the feature the hotspot is associated with. The hot spots in all embodiments may provideaccess into any content regardless of its source so that the hot spotsmay allow trainees to search the web or other sites for information on aparticular aspect of the component with which the hot spot isassociated. The web access may be configured so that only information onthat particular feature is available to the trainee. The system,apparatus, and/or interface may therefore also include a webpre-interface that acts to restrict access except to the material neededby the trainee to complete the training or to complete a given taskassociated with the training routine.

Biological Teaching Embodiment

Referring now to FIGS. 8A-D, an embodiment of a system, apparatus,and/or interface of this disclosure, generally 800, illustrates aprokaryotic versus eukaryotic cells image and a set or plurality of hotspots or active areas used in a training routine.

Looking at FIG. 8A, the system, apparatus, and/or interface is shown toinclude a display device 802 and an image area 804 containing an image806 of a prokaryotic and eukaryotic cell side by side. The system,apparatus, and/or interface has analyzed the image 806 and hot spots a-hare generated associated with the prokaryotic cell and hot spots s-z aregenerated associated with the eukaryotic cell. In an like manner, thehot spots a-h and hot spots s-z are represented by transparent circledletters and are associated with relevant feature of the image 806. Atrainee/student may activate each of the selectable objects using anyselecting procedure including motion, gestures, mouse selecting,touching, etc.

Looking at FIG. 8B, the display area 804 is split into an image displayarea 804 a and a content display area 804 b, when the trainee/studentselects hot spot s associated with the eukaryotic cell and indicated bythe hot spot s changing color or some other attribute of the hot spot s,which also appears highlighted in the content display area 804 b alongwith selectable content objects 1-10 associated with the hot spot s thatinclude relevant information about the hot spot s. The selectablecontent objects 1-10 are shown here distributed about a centralselection object 808, which may or may not be visible. Each one of theselectable objects 1-10 may include information and/or tasks that thetrainee/student must master concerning the geological image 806. Ofcourse, it should be recognized that the selectable content objects 1-10may be distributed in any pattern within the display area 804 b,including a random or ordered pattern and the number of selectablecontent objects may be any number from 1 to 1000 or more depending onthe amount of information that may be associated with each hot spot.

Looking at FIG. 8C, in an alternate embodiment, the system, apparatus,and/or interface changes the color of the hot spot s, when selected anddisplays the selectable content objects 1-10 associated with the hotspot s distributed above the hot spot s. Of course, the selectablecontent objects 1-10 may be distributed in any pattern within thedisplay area 804 b, including a random or ordered pattern. Moreover, thenon-selected hot spot t-z may disappears so that the selectable contentobjects 1-10 may surround the hot spot s. Selection of the selectablecontent objects 1-10 may be by any selection methodology.

Referring now to FIG. 8D, the system, apparatus, and/or interfacegenerates a 3D environment 810 of the eukaryotic cell of the image 806.The system, apparatus, and/or interface also associates the hot spotss-z with their corresponding features in the 3D environment 810. Thetrainee/student may then travel in the 3D environment 810 changingviewpoints, looking at features, activating any of the hot spots s-z andreviewing associated content associated therewith and performing anytasks associated with the hot spots s-z. Again, a trainee may travelsanywhere within the 3D environment. As the trainee travels through theenvironment, object may be expanded and hot spots associated withdifferent aspects of expanded components will change along with the typeof information that is presented upon activating one of the hot spots.Hot spot content may also change based on the level of training and eachhot spot may have training exercises associated with the feature the hotspot is associated with. The hot spots in all embodiments may provideaccess into any content regardless of its source so that the hot spotsmay allow trainees to search the web or other sites for information on aparticular aspect of the component with which the hot spot isassociated. The web access may be configured so that only information onthat particular feature is available to the trainee. The system,apparatus, and/or interface may therefore also include a webpre-interface that acts to restrict access except to the material neededby the trainee to complete the training or to complete a given taskassociated with the training routine.

Computer Repair Embodiment

Referring now to FIGS. 9A-H, an embodiment of a system, apparatus,and/or interface of this disclosure, generally 900, depict an embodimentof a virtual computer assembly training routine. Looking at FIG. 9A, thesystem, apparatus, and/or interface is shown to include a display device902 includes a computer assembly area 904 containing a desktop cabinet906. The desktop cabinet 906 includes a motherboard receiving area 908,a power supply receiving area 910, a fan receiving area 912, a receivingunit 914 having bays 916 for CD drives, DVD drives, floppy drives, etc.,and a receiving unit 918 including bays 920 for receiving hard drives.Each area 918-914 and 918 include hotpots j-n shown as the lettersinside a circle. The hot spots j-n, when activated, show informationabout the areas and how to install their respective components. Thedisplay device 902 also include a computer component area 922 includinga Power Supply, a Fan, a Motherboard, a CPU, a CD/DVD drive, a VideoCard, a Hard Drive, a Sound Card, and RAM. The components include hotspots a-i shown as the letter inside a circle. The hot spots whenactivated discloses information about the components and how they areinstalled and connected to the other components. The training exercisegenerally starts with a video describing the assembly process. Afterwatching the video, the system, apparatus, and/or interface creates aVR/AR/MR environment that allows a trainee to actually assemble thecomputer from the components. While the video may show a specific way,generally an optimal way, to assemble the computer, the trainee is freeto change the order to learn why the video is optimal. The system,apparatus, and/or interface capture data as the trainee adds eachcomponent to the cabinet that then connecting the components together.The system, apparatus, and/or interface analyzes the data and evaluatesthe trainee at each task and provides the trainee with a proficiencyscore. The system, apparatus, and/or interface provides feedback to thetrainee before, during and after each task and provides information asto proficiency allowing the trainee to repeat tasks until the traineeachieves a desired proficiency.

Looking at FIG. 9B, the system, apparatus, and/or interface shows thetrainee has activated the Motherboard content object c causing contentinformation items 1-4 to appear. It should be recognized that the numberof information items will vary and may be dependent on the component andthe training level. These items may be selected using any selectionprotocol mentioned herein, but especially motion-based scrolling andselecting and activating. Of course, the items 1-4 may be distributedabout the object c in any arrangement to provide better selection or theitems 1-4 may appear in a separate window or above or below the plane ofthe displays if the environment is 3D.

Looking at FIG. 9C, the system, apparatus, and/or interface shows thatthe trainee has installed the Motherboard into the motherboard area 908and activated the object e associated with the CD/DVD drive. The hotspots c and 1 remain associated with their respective component or areaso that the trainee may be able to review information about theMotherboard and the area motherboard 908, where the information may belevel specific.

Looking at FIG. 9D, the system, apparatus, and/or interface shows thatactivating CD/DVD drive content object e causing content informationitems 1-10 to appear (note that the number of information items willvary and may be dependent on the component and the training level) inthe Motherboard space now empty as ten selectable objects distributedabout a selection object 950, which may or may not be visible. Theseitems may be selected using any selection protocols mentioned herein,but especially motion-based scrolling and selecting and activating.

Looking at FIG. 9E, the system, apparatus, and/or interface shows thatthe trainee has installed the CPU onto the Motherboard. Again, the hotspot d remains associated with the CPU so that the trainee may be ableto review information about the CPU, where the information may be levelspecific. The figure also shows that the trainer has activated the PowerSupply object a.

Looking at FIG. 9F, the system, apparatus, and/or interface shows thetrainee activating Power Supply content object a causing contentinformation items 1-10 to appear (note that the number of informationitems will vary and may be dependent on the component and the traininglevel) in the Motherboard space now empty as ten selectable objectsdistributed about a selection object 950, which may or may not bevisible. These items may be selected using any selection protocolsmentioned herein, but especially motion-based scrolling and selectingand activating.

Looking at FIG. 9G, the system, apparatus, and/or interface shows thatthe trainee has installed the RAM onto the Motherboard, the CD/DVD drivein a bay 916 of the receiving unit 914, and two Hard Drives in two bay920 of the receiving area 918, and the Fan into the fan area 912. Again,the hot spot i the hot spot b remains associated the RAM so that thetrainee may be able to review information about the RAM, the hot spot ewith the CD/DVD drive so that the trainee may be able to reviewinformation about the CD/DVD drive, the hot spot g with the Hard Drivesso that the trainee may be able to review information about the HardDrives, and the hot spot b with the Fan so that the trainee may be ableto review information about the Fan, where the information may be levelspecific. The figures also shows that the trainer has activated thePower Supply object a. Of course, it should be recognized that theinstallation of these components may have been accomplished by anyorder, so that the trainee may understand why the optimal assembly orderwas shown in the training video.

Looking at FIG. 9H, the system, apparatus, and/or interface shows thatthe trainee has installed the Video Card and the Sound Card onto theMotherboard. Again, the hot spot f remains associated the Video Card sothat the trainee may be able to review information about the Video Card,and the hot spot h with the Sound Card drive so that the trainee may beable to review information about the Sound Card, where the informationmay be level specific. Of course, it should be recognized that theinstallation of these components may have been accomplished by anyorder, so that the trainee may understand why the optimal assembly orderwas shown in the training video.

Referring now to FIG. 9I depict the system, apparatus, and/or interfacegenerates a 3D environment 924 based on an assembled desktop computer.The system, apparatus, and/or interface also generated hot spots a-u andassociates them with different features of the assembled desktopcomputer shown in 3D environment 924. The trainee/student may thentravel in the environment 924 changing viewpoints, looking at features,activating any of the hot spots a-u and reviewing and performing anytasks associated with the hot spots. The trainee may repeat any and alltasks to until the trainee has achieved a given degree of overallproficiency and a given degree of proficiency or each task associatedwith the training routine. Again, the system, apparatus, and/orinterface is designed to collect/capture trainee data and expert data asthey interact with the training environment. Again, the system,apparatus, and/or interface stores the data in system databases locallyor remotely. Again, the system, apparatus, and/or interface uses thedata to improve the training environment and to improve hot spot contentand the manner in which the trainer or expert interact with the overalltraining routine and each task within the training environment.

Schematic Flow Charts Virtual Training Methods

Referring now to FIG. 10, a schematic flow chart of an embodiment of amethod of training, generally 1000, is shown to include a start step1002. The methods also include identifying a non-animated training task,routine, or program for automation using the systems, apparatuses,and/or interfaces herein in a identifying step 1004. The methods alsoinclude developing a VR or AR/MR/XR training routine corresponding tothe in a developing step 1006.

The methods may also include generating an avatar assistant in an avatargenerating step 1008, wherein the avatar may change form depending onthe task, may be generated using user input, may be generated based onpast trainee performance, or may adapt a specific form to improvetrainer environment and routine interaction. The methods also include:(a) generating information about all important, relevant, meaningful,and/or interesting features, attributes, properties, and/orcharacteristics associated with the training routine and all important,relevant, meaningful, and/or interesting features, attributes,properties, and/or characteristics associated with any and all equipmentused in the training routine or associated with the training routineand/or (b) gathering information about all important, relevant,meaningful, and/or interesting features, attributes, properties, and/orcharacteristics associated with the training routine and all important,relevant, meaningful, and/or interesting features, attributes,properties, and/or characteristics associated with any and all equipmentused in the training routine or associated with the training routine inan information generating and gathering step 1010. The methods alsoinclude storing the gathered information in system databases bothlocally and/or remotely, e.g., databases in the cloud or dedicatedservers in a storing step 1012.

The methods also include analyzing the training routine, generating aVR/AR/MR environment for training, and associating/populating hot spotswith all relevant, meaningful, or interesting features, attributes,properties, and/or characteristics associated with the trainingenvironment and all important, relevant, meaningful, and/or interestingfeatures, attributes, properties, and/or characteristics associated withany and all equipment used in the training environment or associatedwith the training routine in an analyzing, environment generating, andhot spot associating step 1014.

The methods also include collecting/capturing expert data as an expertgoes through the training routine in the training environment, whereinthe data includes data on how the expert accomplished each trainingtask, data on what hot spots the trainee activates, data on the durationof time the expert spends in teach hot spot and data on the type ofinformation viewed, data indicating a level of competency in each task,data indicating an overall competency, and data associated with ease ofusing each task, ease of using the entire training routine, and each ofusing and interacting with the environment in a collecting/capturingexpert performance data step 1016. The methods also include processingthe expert performance data, overall data and task data and storing thedata in the system databases in a processing and storing expert datastep 1018. The methods may also include an expert feedback step forproviding experts with overall performance and task performance data togain insight from experts on how to improve the training routine andenvironment (not shown).

The methods also include collecting/capturing trainee data as a traineegoes through the training routine in the training environment, whereinthe data includes data on how the trainee accomplished each trainingtask, data on what hot spots the trainee activates, data on the durationof time the trainee spends in teach hot spot and data on the type ofinformation viewed, data indicating a level of competency in each task,data indicating an overall competency, and data associated with ease ofusing each task, ease of using the entire training routine, and each ofusing and interacting with the environment in a collecting/capturingtrainee performance data step 1020.

The methods also include providing feedback to the trainee and/or to thesupervisors before, during, and/or after each task or before, during,and/or after the entire training routine to the trainee and collectingand storing all feed back data and evaluation data made by supervisorsin a providing trainee/supervisor feedback step 1022. The methods arecontinued onto the next page via a next step 1024.

After the next step 1024, the methods also include collecting/capturingand storing the feedback data to the system databases in acollecting/capturing and storing the feedback data step 1026. Themethods also include informing the trainee of any deficiencies andproviding information on how to improve the trainees performanceincluding showing the trainees performance overlaid on an expertsperformance for the same task or suggesting way to improve performancein an informing trainee deficiency step 1028. The methods also includeproviding information to the trainee to improve performance deficienciesin a providing trainee information step 1030.

The methods also include repeating the deficient tasks or the entireroutine until proficient at each task and the entire routine to a givenlevel of competency in a trainee repeating step 1032. The methods alsoinclude providing feedback to the trainee/supervisor before, during,and/or after the trainee repeats routine/tasks in a providing feedbackon repeated routine/tasks step 1034. The methods also include processingand storing the trainee repeat performance data in the system databasesin a processing and storing step 1036.

The methods also include analyzing performance data and modifying one,some, or all aspects of the training routine, task, avatar, hot spot,and/or environment features, properties, characteristics, attributes,etc. in a analyzing and modifying identified problem tasks and/oraspects step 1038 to improve the routine, tasks, avatars, hot spots,and/or environment.

The methods also include analyzing performance data (expert and trainee)and modifying one, some, and/or all aspects of the routine, the task,the environment, the avatar, and/or the hot spot features, properties,characteristics, attributes, etc. are modified to improve any and allaspects of the training routine, the tasks, the environment, theavatar(s), and/or the hot spots to aid the trainee in those deficienttasks and allowing the trainee to redo the training task or the entiretraining session in an analyzing and modifying step 1040.

The methods also include continuing the analyzing, determining, andimproving the routine, environment, avatar, hot spots, and/or hot spotcontent, wherein the improving may include revising, adding, deleting,modifying, and/or changing any some or all aspects of the trainingroutine, environment, avatar, hot spots, and/or hot spot content toenhance its effectiveness over time in a continually analyzing andmodifying step 1042. The methods terminates in a stop step 1044.

Schematic Flow Chart for Developing a VR and/or AR/MR/XR TrainingProgram

Referring now to FIG. 11, a schematic flow chart of an embodiment of amethod of constructing a VR and/or AR/MR/XR training program from apre-existing training program, generally 1100, is shown to include astart step 1102.

The training program may include, without limitation, one, some, all orany combination of the following components: device explanatoryroutines, training routines, training tasks, question/answer sessions,information presentation routines, trainee pass/fail routines,performance ranking routines, trainee feedback routines, or etc.

The methods include loading a pre-existing non VR or AR/MR/XR trainingprogram into the systems, apparatuses, and/or interfaces of thisdisclosure in a loading step 1104. The methods also include: (a)generating information about one, some, or all aspects, features,attributes, properties, and/or characteristics associated with thetraining program and/or one, some, or all aspects, features, attributes,properties, and/or characteristics associated with any and all equipmentused in the training program or associated with the training programand/or (b) gathering information about one, some, or all aspects,features, attributes, properties, and/or characteristics associated withthe training program and one, some or all aspects, features, attributes,properties, and/or characteristics associated with any and all devicesand/or equipment used in the training program or associated with thetraining program in an information generating and/or gathering step1106. The methods also include storing the generated and/or gatheredinformation in system databases both locally and/or remotely, e.g.,databases located on dedicated servers locally or on dedicated remoteservers such as cloud servers serves located in the cloud, in a storingstep 1108.

The methods also include analyzing the pre-existing training program,the generated information, and the gathered information in an analyzingstep 1110 and developing VR content, AR/MR/XR content, and/or VR andAR/MR/XR content for the pre-existing training program in a developingstep 1112. The VR content and/or AR/MR/XR content may be generated fromthe training material in whatever form the content is found. If thecontent is purely textual, then the systems, apparatuses, and/orinterfaces analyze the textual information and prepare CG constructsbased on the textual content. The CG constructs may be VR and/orAR/MR/XR depending on the nature of the textual content and the best waythe systems, apparatuses, and/or interfaces determine the content may bepresented to the trainees, which may be modified during expert andtrainee performance of the training program. If the content includestextual and non-textual (e.g., images, figures, videos, etc.), then thesystems, apparatuses, and/or interfaces analyze the textual andnon-textual content and prepare CG constructs based on the textual andnon-textual content. Again, the CG constructs may be VR and/or AR/MR/XRdepending on the nature of the textual and the non-textual content andthe best way the systems, apparatuses, and/or interfaces determine thecontent may be presented to the trainees, which may be modified duringexpert and trainee performance of the training program. If the contentis purely non-textual, then the systems, apparatuses, and/or interfacesanalyze the non-textual content and prepare CG constructs based onnon-textual content. Again, the CG constructs may be VR and/or AR/MR/XRdepending on the nature of the training program content and the best waythe systems, apparatuses, and/or interfaces determine the content may bepresented to the trainees, which may be modified during expert andtrainee performance of the training program.

If the textual and/or non-textual content relate to one or more devicesor one or more pieces of equipment, then the systems, apparatuses,and/or interfaces may construct 2D, 3D, 4D, or nD CG constructsdepicting the devices or equipment so that the trainee if interestedcould see how the devices or equipment were constructed and/or how theyoperate. Of course, the systems, apparatuses, and/or interfaces makesthis content available to a user as an selectable object or a hot spotassociated with environment such as a thumbnail of a device or a pieceof equipment or a document including information about the device orequipment.

If the training program involves a particular device or piece ofequipment, then the VR and/or AR/MR/XR environment would include alltypes of content associate with the device or piece of equipment andwould include all types of formats (e.g., documents, images, videos,recordings, drawings, hot spots, etc.) for interacting with the contentincluding VR and/or AR/MR/XR CG constructs that a trainee interacts withand that behaves as the real device or piece of equipment such as aninstrument would behave in physical reality. Such VR and/or AR/MR/XRenvironments include, without limitation, flight simulations, games,unmanned aerial vehicles (UAV), unmanned nautical vehicles, unmannedground based vehicles, computer aided surgeries, computer aidedinstruments, computer aided manufacturing, all other computer aidedhuman endeavors, or any combination thereof.

The methods also include generating a VR environment, an AR/MR/XRenvironment, or a VR and an AR/MR/XR environment for the trainingprogram in a generating step 1114. The environments are designed to befully interactive based primarily on motion-based processing, but theenvironment may also include the ability for the user to interact withthe environment using: hard select protocol input such as mouseclicking, touching events on a touchscreen or touch pad, button pushing,tactile switches, etc. or voice or utterance input or gesture input orany combination of motion-based, hard select protocol based, utterancebased, or gesture based inputs.

The methods also include generating hot spots for one, some or allaspects, features, attributes, properties, and/or characteristicsassociated with the training environment in a hot spot generating step1116, and populating the environment with the generated hot spots in apopulating step 1118. The hot spots may be associated with (a) specificaspects, features, attributes, properties, and/or characteristics of thetraining program or (b) may be general information hot spots associatedwith global environmental aspects, features, attributes, properties,and/or characteristics or (c) aspects, features, attributes, properties,and/or characteristics of training program routines and/or trainingprogram tasks. For example, the global aspects, features, attributes,properties, and/or characteristics may include color, lighting,highlighting, day or night, brightness, volume, etc. and the hot spotswould provide the user access to these global aspects, features,attributes, properties, and/or characteristics so that the user maypreset them. Other hot spots may allow the user to preset aspects,features, attributes, properties, and/or characteristics with specifictraining program routines and/or tasks.

In certain embodiments, the methods may be divided into parts. The firstpart involves the construction of a VR environment, an AR/MR/XRenvironment or a VR and an AR/MR/XR environment for a given trainingprogram. The second part involves having experts perform the trainingprogram so that the training program, environment, content, hot spots,and/or avatars may be modified to improve the training program. Thethird part involves having trainees perform the training program so thatthe training program, environment, content, hot spots, and/or avatarsmay be modified to improve the training program. The fourth partinvolves using trainer and trainee data to tailor the training programfor different trainee types or for trainees having different learningpropensities oral, visual, tactile, etc.

The methods also include a next step 1120 for continuing the flow chartto the next drawing sheet.

After construction of the environment, the methods also include startingan expert training session in a expert or trainer start step 1122.

The methods also include collecting/capturing expert/trainer performanceand interaction data as an expert/trainer performs the training programin the constructed training environment in a collect/captureexpert/trainer data step 1124. The expert/trainer performance andinteraction data includes: (v) data on how the expert accomplishes theentire program, (w) data on how the expert accomplishes each routineand/or task of the program, (x) data on what hot spots the expertactivates, (y) data on the duration of time the expert spends in eachactivated hot spot, (z) data on the type of information viewed, (aa)data indicating a level of competency in entire program and each routingand/or task, (bb) data associated with ease of performing each of theroutine and/or task, (cc) data associated with the ease/difficulty ofusing the entire training program, (dd) data associated with theease/difficulty of using each routine or task, (ee) data associated withusing and interacting with the entire environment, and/or (ff) dataassociated with using and interacting with one, some, or all aspects,features, attributes, properties, and/or characteristics of the program,environment, routines, tasks, content, and/or hot spots, and/or (gg)data that may be used to construct predictive tools or routines fortailoring the training program to individual trainees.

The methods also include processing the expert/trainer performance andinteraction data and storing the expert/trainer performance andinteraction data in the system databases in an expert/trainer dataprocess and store step 1126. The processing may entail formatting theinformation for more efficient database storage and retrieval. Themethods also including analyzing the collected/captured performance andinteraction data in an analyzing step 1128. The methods also includemodifying one, some, or all aspects, features, attributes, properties,and/or characteristics of the training program, routines, tasks,environment, and/or hot spots, during training to improve the program ina modifying step 1130. The methods also include updating the program andstoring the updated program the system databases in an updating step1132.

The methods may also include providing feedback to the expert withoverall performance and task performance data and interaction to gaininsight from experts on how to improve the training program andenvironment in an expert feedback step 1134. The methods also include aconditional step to permit the expert to repeat the training session orspecific routines or tasks therein based on the feedback in a satisfiedstep 1136. If the expert is not satisfied with any aspect of theperformance, then the methods proceed along a NO branch to thecollecting/capturing step 1124 so that the expert repeats the entireprogram or routines or tasks thereof repeating all intermediate steps.The expert may repeat the program or any task thereof as many times asdesired until the expert is satisfied at which point the methods proceedalong a YES branch to a continuation step 1138 to continue to the nextpage of the drawings.

The methods may also include generating one or more avatars in an avatargenerating step 1140 and providing the avatar with the expertperformance and interaction data in a providing step 1142. The systems,apparatuses, and/or interfaces may construct any number of avatars,wherein the avatars may be constructed for specific routines or tasks.Of course, the systems, apparatuses, and/or interfaces may construct anynumber of avatars, wherein the avatars may be constructed for specificroutines or tasks.

After an expert performs the training session and the systems,apparatuses, and/or interfaces have been modified based on the expertperformance and interaction data, the methods include a start traineetraining session in a starting trainee step 1144. In certainembodiments, the methods may actually start here as the expert sessionand modifying steps may proceed independent of any trainee activity asthe environment constructing may proceed independently.

The methods also include collecting/capturing trainee performance andinteraction data as a trainee goes through the training program in thetraining environment, wherein the performance and interaction dataincludes data on how the trainee accomplishes the entire program, dataon how the trainee accomplishes each task of the program, data on whathot spots the trainee activates, data on the duration of time thetrainee spends in each activated hot spot, data on the type ofinformation viewed, data indicating a level of competency in entirerouting and each task, and/or data associated with ease of performingeach of the routines or tasks, ease/difficulty of using the entiretraining program and ease/difficulty of using each routine or taskthereof, and data associated with using and interacting with the entireenvironment and/or one, some, or all aspects, features, attributes,properties, and/or characteristics of the environment, routines, tasks,content, and/or hot spots in a collecting/capturing trainee performanceand interaction data step 1146.

The methods also include processing the trainee performance andinteraction data and storing the trainee performance and interactiondata in the system databases in a processing and storing trainee datastep 1148, wherein the processing may entail formatting the informationfor more efficient database storage and retrieval. The methods alsoincluding analyzing the collected/captured trainee performance andinteraction data in an analyzing step 1150.

The methods also include modifying one, some, or all aspects, features,attributes, properties, and/or characteristics of the environment,routines, tasks, content, and/or hot spots of the training program,routines, tasks, environment, hot spots, and/or avatars during trainingto improve the program in a modifying step 1152. The methods alsoinclude updating the program and storing the updated program the systemdatabases in an updating step 1154. The methods also include providingfeedback to an expert/supervisor before, during, and/or after each taskor before, during, and/or after the entire training program of a traineein a providing expert/supervisor feedback step 1156. The methods alsoinclude providing feedback to the trainee including deficient taskperformance data in a providing trainee feedback step 1158.

The methods also include a conditional step to permit the trainee torepeat the training session or specific routines or tasks therein basedon deficient task feedback in a pass step 1160. If the trainee does notpass the training program or any routine or task or aspect of thetraining program, then the methods proceed along a NO branch to thecollecting/capturing step 1146 so that the trainee may repeat the entireprogram or deficient routines or tasks or aspects thereof and repeatingall intermediate steps. The trainee may repeat the program or any taskthereof as many times as needed to pass the training program or anyrouting or task or aspect at which point the methods proceed along a YESbranch to a continuation step 1162 for continuing to a next page of thedrawings.

The methods also include processing the trainee pass performance andinteraction data and storing the trainee pass performance andinteraction data in the system databases in a processing and storingexpert data step 1164, wherein the processing may entail formatting theinformation for more efficient database storage and retrieval. Themethods also including analyzing the trainee pass performance andinteraction data in an analyzing step 1166. The methods also includemodifying one, some, or all aspects of the training program, routines,tasks, environment, hot spots, and/or avatars during training to improvethe program in a modifying step 1168. The methods also include updatingthe program and storing the updated program the databases in an updatingstep 1170.

The methods also include continuously, periodically, or intermittentlyanalyzing all of the stored data in an analyze step 1172. The methodsalso include developing predictive tools or routines to assist alltrainees in performing the training program and/or navigating theenvironment in a develop global predictive tool step 1174 and developingtrainee type predictive tools or routines to assist specific types oftrainees in performing the training program and/or navigating theenvironment in a develop trainee type predictive tool step 1176. Themethods also include updating the program, routines, tasks, environment,hot spots, and/or avatars and storing the updates in the update andstore step 1178. The methods then stop in a stop step 1180.

It should be recognized that the above steps relating to developingpredictive tools or routines may be applied to the methods of FIG. 10.

Modifying a Cockpit Virtual Training Program Based on User Interaction

Referring now to FIG. 12A, an embodiment of a system, apparatus, and/orinterface of this disclosure, generally 1200, is shown to include adisplay device 1202 including a display area 1204 containing an airplanecockpit 1206, which may be a 2D photographic image or a 3D CG construct.The system, apparatus, and/or interface then: (v) analyzes the airplanecockpit 1206, (w) identifies components for which hot spot informationmay be generated, (x) generates hot spots a-q, represented bytransparent encircled letter selectable objects, for the identifiedcomponents, and (y) populating the airplane cockpit 1206 with the hotspots a-q, represented by the transparent encircled letter selectableobjects. A trainee may activate each of the hot spots a-q via theencircled representative selectable objects using any selectingprocedure including motion, gestures, mouse selecting, touching, etc.

Referring now to FIG. 12B, during trainee interaction with the cockpit1206, the trainee identified four components for which the traineewanted additional information indicated by transparent circlesassociated with the indicated components.

Referring now to FIG. 12C, the cockpit 1206 has been modified to add newhot spots r-u represented by orange encircled letters r-u. Upon system,apparatus, and/or interface updating, the cockpit 1206 would now includehot spots a-u.

Referring now to FIG. 12D, after collecting data of trainees interactingwith the cockpit 1206 over a period of time with, the system, apparatus,and/or interface may prioritize the hot spots a-u either via color,size, or other hot spot characteristic. In this figure, hot spots e, h,k, m, and n have changed size. Hot spots a-e, h, 1, and n are colorcoded. Hot spots a-d have been ranked by color shading. Of course, anyranking or prioritizing format may be used.

Interacting with a Cockpit Virtual Training Routine Based on User SensedMotion

Referring now to FIGS. 13A-J, an embodiment of a system, apparatus,and/or interface of this disclosure, generally 1300, is shown here inFIG. 13A to include a display device 1302 including an image area 1304containing an airplane cockpit 1306, which may be a 2D photographicimage or a 3D CG construct. The cockpit 1306 includes hot spots a-q,represented by transparent encircled letter a-q selectable objects.

Looking at FIG. 13B, the system, apparatus, and/or interface receivesinput from motion sensor indicated by an arrow extending from aselection object 1308. It should be recognized that the system,apparatus, and/or interface may not actually display a selection objector an arrow, the system, apparatus, and/or interface may simply jump toFIG. 13D indicating the selection of hot spot 1 based on motion datareceived by the system, apparatus, and/or interface from the motionsensor.

Looking at FIG. 13C, based on the motion, the system, apparatus, and/orinterface expands the cockpit 1306 so that further motion aids in hotspot discrimination. The motion sensor then detects further motion ofthe trainee indicated by an arrow extending from the selection object1308, which sufficient to identify the hot spot 1 as the target, whichhas become highlighted in red.

Looking at FIG. 13D, upon the identification of hot spot 1, the system,apparatus, and/or interface clears the display area 1304 and displays anencircled red highlight 1 icon 1310 and the selection object 1308.Surrounding the selection object 1308, the system, apparatus, and/orinterface displays a documents box 1312, a videos box 1314, asimulations box 1316, an interact with experts box 1318, an expertanalyses box 1320, a tutorials box 1322, and an image 1324.

Looking at FIG. 13E, the system, apparatus, and/or interface receivesmotion input from the motion sensor that the user has moved toward thedocuments box 1312 resulting in the highlighting of the documents box1312 and the display of a Doc1 box 1312 a, a Doc2 box 1312 b, a Doc3 box1312 c, a Doc4 box 1312 d, and a Doc5 box 1312 e.

Looking at FIG. 13F, based on analysis of trainee interaction data, thesystem, apparatus, and/or interface determined that additionaldocumentary information needs to be added to the documents box 1312 asnew boxes, a Doc6 box 1312 f and a Doc7 box 1312 g. The new documentsmay be specifically directed to specific information based on thetrainee interaction data. The system, apparatus, and/or interface wouldthen update and store the updates.

Looking at FIG. 13G, the system, apparatus, and/or interface receivesinput from motion sensor indicated by an arrow extending from theselection object 1308 in a direction that may indicate the selection ofeither the documents box 1312 or the images box 1324 causing the twoboxes 1312 and 1324 highlighted and documents and images associated withthe boxes 1312 and 1324. The documents associated with the documents box1312 include the Doc1 box 1312 a, the Doc2 box 1312 b, the Doc3 box 1312c, the Doc4 box 1312 d, and the Doc5 box 1312 e. The images associatedwith the images box 1324 include an Img1 box 1324 a, an Img2 box 1324 b,an Img3 box 1324 c, an Img4 box 1324 d, an Img5 box 1324 e, an Img6 box1324 f, and an Img7 box 1324 g.

Looking at FIG. 13H, in further response to the motion, the system,apparatus, and/or interface reorganizes the boxes 1312, 1314, 1316,1318, 1320, and 1322 so that the two aligned boxed 1312 and 1324 areseparated and the other boxes are moved away, here to the bottom of thearea 1304. Here the two box 1312 and 1324 are display in the center ofthe area 1304, with the box 1324 to the left and the box 1312 to theright. The figure also shows that the system, apparatus, and/orinterface has received further motion data from the motion sensorindication motion toward the images box 1324 causing the system,apparatus, and/or interface to remove the highlighting on the documentsbox 1312 indicating the selection of the images box 1324.

Looking at FIG. 13I, in response to the further motion, the system,apparatus, and/or interface moves the documents box 1312 away, here tothe bottom of the area 1304 with the other boxes and centers the imagesbox 1324 with the selection object 1308 above the images box 1324 anddisplays the Img1 box 1324 a, the Img2 box 1324 b, the Img3 box 1324 c,the Img4 box 1324 d, the Img5 box 1324 e, the Img6 box 1324 f, and theImg7 box 1324 g in a radially spaced apart configuration about theselection object 1308. The system, apparatus, and/or interface alsoreceives yet further motion data from the motion sensor as shown by thearrow extending from the selection object 1308 towards two image boxes,Img2 box 1324 b and Img3 box 1324 b, which are highlighted.

Looking at FIG. 13J, in response to the yet further motion, the system,apparatus, and/or interface moves the Img1 box 1324 a, the Img4 box 1324d, the Img5 box 1324 e, the Img6 box 1324 f, and the Img7 box 1324 gaway from and centers the Img2 box 1324 b and the Img3 box 1324 c. Thesystem, apparatus, and/or interface receives input from motion sensorindicated by an arrow extending from the selection object 1308 in adirection that the Img2 box 1324 b is selected.

The above illustrated hot spot selection, activation and modification,although shown in a different display area, may in fact be superimposedup the image or generated within the 3D environments.

Avatar Head and Face Expressions Utilization to Aid in Training

Referring now to FIGS. 14A-T, an embodiment of a system, apparatus,and/or interface of this disclosure, generally 1400, is shown to includea display area 1402 displaying a CG avatar head 1404 having a variety offacial expressions, wherein the facial expressions are designed toexpress to the trainee different reactions to user performance.

Looking at FIGS. 14A-C, three different smiling avatar facialexpressions. Each smiling facial expression may indicate differentdegrees of approval. For example, the three smiling expressions mayindicate that the systems, apparatuses, and/or interfaces are (a)somewhat happy with the trainee performance as evidenced by downwardslanting eyebrows, (b) happy with the trainee performance as evidence bystraight eyebrows, and (c) very happy with the trainee performance asevidence by upward slanting eyebrows. It should be recognized thatdifferences in the degree of approval as evidence by the smile may alsobe evidenced by changing the depth of the smile or may any combinationof changes in the smiling facial expression to indicate differentdegrees of approval or success in performing the entire training programor any aspect of the training program.

Looking at FIGS. 14D-F, three different straight avatar facialexpressions. Each straight facial expression may indicate differentdegrees of approval. For example, the three straight facial expressionsmay indicate somewhat below average trainer performance as evidenced bydownward slanting eyebrows, average trainee performance as evidence bystraight eyebrows, and above average trainee performance as evidenced byupward slanting eyebrows. Again, it should be recognized that anycombination of changes in the straight facial expression may indicatedifferent degrees of approval or success in performing the entiretraining program or any aspect of the training program.

Looking at FIGS. 14G-I, three different frowning avatar facialexpressions. Each frowning expression may indicate different degrees ofdisapproval. For example, the three frowning expressions may indicate avery negative trainee performance as evidenced by downward slantingeyebrows, a negative trainee performance as evidence by straighteyebrows, and somewhat negative trainee performance as evidenced byupward slanting eyebrows. Again, it should be recognized that anycombination of changes in the frowning facial expression may indicatedifferent degrees of disapproval in performing the entire trainingprogram or any aspect of the training program.

Looking at FIGS. 14J-L, three different surprised avatar facialexpressions. Each surprised expression may indicate different degrees ofsurprise. For example, the three surprised expressions may indicatesomewhat surprised to user performance as evidenced by downward slantingeyebrows, surprised to user performance as evidence by straighteyebrows, and very surprised to user performance as evidenced by upwardslanting eyebrows. Again, it should be recognized that any combinationof changes in the surprised facial expression may indicate differentdegrees of disapproval in performing the entire training program or anyaspect of the training program.

Looking at FIGS. 14M-P, four different wide-eyed avatar facialexpressions. Each wide-eyed expression may indicate different responsesto trainee performance. Here, four wide-eyed expressions are shown: (a)a straight mouth and straight eyebrow expression, (b) a smiling mouthand raised eyebrow expression, (c) a frowning mouth and straight eyebrowexpression, and (d) a surprised mouth and lowered eyebrow expression.Each of these expressions may appears after a trainee performs theentire program or at any stage of performing any aspect of the trainingprogram.

Looking at FIGS. 14Q-T, four different narrow-eyed avatar facialexpressions. Each narrow-eyed expression may indicate differentresponses to trainee performance. Here, four narrow-eyed expressions areshown: (a) a straight mouth and straight eyebrow expression, (b) asmiling mouth and lowed eyebrow expression, (c) a frowning mouth andraised eyebrow expression, and (d) a surprised mouth and straighteyebrow expression. Each of these expressions may appears after atrainee performs the entire program or at any stage of performing anyaspect of the training program. and very surprised to user performance.

Avatar Body Posture Utilization to Aid in Training

Referring now to FIGS. 15A-F, embodiments of a system, apparatus, and/orinterface of this disclosure, generally 1500, is shown to include a CGavatar body having a variety of body expressions. Each body expressionmay be combined with any of the facial expressions and may be used toindicate different system responses to trainee performance of the entiretraining program or any aspect therefore.

Looking at FIGS. 15A-C, three body expressions are shown here: (a) abody expression with arms raised, (b) a body expression with one armraised and one arm lowered, and (c) a body expression with both armlowered. Again, the CG body may have other features and all of thefeatures including the facial features may change to indicate differentresponses to trainee performance of the entire training program or anyaspect thereof and may change at any time during trainee interactionwith the training program.

Looking at FIGS. 15D-F, three bouncing body expressions are shown here:(a) a bouncing body expression with arms raised, (b) a bouncing bodyexpression with one arm raised and one arm lowered, and (c) a bouncingbody expression with both arm lowered. Again, the CG body may have otherfeatures and all of the features including the facial features maychange to indicate different responses to trainee performance of theentire training program or any aspect thereof and may change at any timeduring trainee interaction with the training program. Additionally, theCG body may bound, dance, spin, or perform any activity to evidence asystem response to any trainee performance.

It should the recognized that the avatars of this disclosure aredesigned to assist and encourage trainees in performing the trainingprogram or any aspect of the training program. Thus, the systems,apparatuses, and interfaces of this disclosure by include avatars thatare capable of expressing all manner of expression or conduct oractivity that will assist a trainee in performing the entire trainingprogram or any aspect thereof, wherein all avatar expressions, conduct,and/or activities are designed to assist and encourage the trainee tosuccessfully accomplish the entire training program and any and allaspects thereof.

Capturing Images, Showing Constructs Creation, and Performing a Routine

Referring now to FIGS. 16A-AG, systems, apparatuses, interfaces, andimplementing methods of this disclosure illustrating the starting acapture session, selecting a desired super hero character, creating a CGconstruct for the super hero proportioned to the user, and capturing theuser performing a series of movements or posture changes and reproducingthe movements or posture changes in the construct. These user constructsmay then be scaled to a set of movements performed in a super hero movieby an actor with differences illustrated so that the user can adjustuser movements until the differences between the user movements and theactor movements are at or below minimum difference criteria.

Looking now at FIG. 16A, the system or apparatus 1600 comprises a touchsensitive screen 1602, which activates when the user 1604 getsufficiently closes to the apparatus 1600. Once activated, the screen1602 displays a glowing orb selection object 1606 and three super heroselectable objects 1608 a-c, along with informational text.

Looking now at FIGS. 16B-G, the user 1604 touches the selection object1606 and illustrates moving the selection object 1606 towards thedifferent super hero objects 1608 a-c until the user 1604 decides toselect a particular super hero 1608 b, where selection occurs either mymoving fully into the object 1608 b or lifting off from the screen atthe object 1608 b.

Looking now at FIGS. 16H-AG, the user 1604 steps back from the screen1602 placing the apparatus 1600 in image captures mode. FIGS. 16I-AGcaptures a set of images of the user 1604 performing a set of movementsrepresenting changing in body postures. The apparatus 1600 then createsa super hero construct 1610 and displays it on the screen 1602 within aAR 3D environment. The construct 1610 is scaled to the user'sproportions and performs the same movements or posture changes displayedwithin a AR 3D environment in response to the user's movements and bodyposture changes. Looking now at FIGS. 16Q-AG, the AR 3D environment hasbeen changed based on the user 1604 user selection from a list ofenvironments (not shown) or by the user looking at a particular scene,here the environment displays the construct 1610 in front of a movietheater. While the present above figures illustrate user image capture,construct and AR generation, and environment selection, if the user isperforming a training routine, then the user construct may be scaled andsuperimposed on the user construct so that differences may be shown andthe user movement may be repeated until the differences are at or belowcertain minimum difference criteria.

Closing

All references cited herein are incorporated herein by reference for alllegal uses including claim support. Although the disclosure has beendisclosed with reference to its preferred embodiments, from reading thisdescription those of skill in the art may appreciate changes andmodification that may be made which do not depart from the scope andspirit of the disclosure as described above and claimed hereafter.

1. A method implemented on an electronic device comprising a processcoupled to or associated with a motion sensor, an input device, adisplay device and an output device, the method comprising: loading apre-existing VR training program comprising a plurality of routines,tasks, or combinations thereof or a AR/MR/XR training program comprisinga plurality of routines, tasks, or combinations thereof; gatheringinformation about: (a) training program features, attributes,properties, characteristics, or combinations thereof including: (b)routine features, attributes, properties, characteristics, orcombinations thereof, and (c) task features, attributes, properties,characteristics, or combinations thereof; (d) program equipmentfeatures, attributes, properties, characteristics, or combinationsthereof associated with equipment used in the pre-existing trainingprogram, routines, and tasks; analyzing the gathered information;creating content from the analyzed gathered information, the contentcomprising program content, routine content, task content, and equipmentcontent; generating hot spots or interactive areas from the createdcontent, the hot spots or interactive areas comprising: (a) hot spots orinteractive areas associated with the program content; (b) hot spots orinteractive areas associated with the routine content; (c) hot spots orinteractive areas associated with the task content; (d) hot spots orinteractive areas associated equipment content; populating thepre-existing VR training program or the pre-existing VR/AR/MR trainingprogram with the generated hot spots or interactive areas; capturingtrainer performance data as a trainer performs the pre-existing VRtraining program or the pre-existing VR/AR/MR training program, thetrainer performance data comprising trainer program competency data,trainer routine competency data, trainer task competency data, trainerhot spot or interact area interaction data, trainer equipmentinteraction data, or any combination thereof; capturing traineeperformance data as a trainee performs the generated training program,the trainee performance data includes trainee program competency data,trainee routine competency data, trainee task competency data, traineehot spot or interact area interaction data, equipment interaction data,or any combination thereof; comparing the trainer performance data andthe trainee performance data; determining differences between thetrainer performance data and the trainee performance data; testing thedifferences between the trainer performance data and the traineeperformance data, if each of the differences exceeds one or more minimumdifference criteria, then: displaying the differences; and for each ofthe routines or tasks having differences that exceeds one or moreminimum difference criteria, repeating: the trainee capturing step, thecomparing step, the determining differences step; the testing step;until each of the differences is less than or equal to the one or moreminimum difference criteria; and indicating successful completion of thetraining program. 2-51. (canceled)
 52. The method of claim 1, furthercomprising: storing the gathered information on local databases, remotedatabases, or both; storing the analysis on the local databases, remotedatabases, or both storing the generated VR training program or AR/MR/XRtraining program on local databases, remote databases, or both; storingthe trainer performance data on the local databases, remote databases,or both; storing the trainee performance data on the local databases,remote databases, or both; storing the differences on the localdatabases, remote databases, or both; providing the trainer with to thetrainer and/or a supervisor with the trainer performance data or anypart thereof; during and/or after the trainee performs the generated VRtraining program or the generated AR/MR/XR training program, providingthe trainee performance data and the differences to the trainer and/orthe supervisor; and during and/or after the trainee performs therepeating step, providing the trainee performance data and thedifferences to the trainer and/or the supervisor.
 53. The method ofclaim 1, further comprising: highlighting the differences according to ahighlighting format, wherein the highlighting format comprising visuallyhighlighting the differences, haptic highlighting the differences via ahaptic device, audio highlighting the differences via a audio device,neurofeedback highlighting the differences via a neurofeedback device,or any combination thereof.
 54. The method of claim 1, furthercomprising: before, during, and/or after the repeating step,illustrating to the trainee how to adjust the trainee's body and/or anypart thereof in the overlaid construct to improve trainee performance.55. The method of claim 1, wherein values of the one or more minimumdifference criteria are: less than or equal to a 20% difference betweenthe trainer performance data and the trainee performance data; less thanor equal to a 10% difference between the trainer performance data andthe trainee performance data; less than or equal to a 5% differencebetween the trainer performance data and the trainee performance data;or less than or equal to a 1% the difference between the trainerperformance data and the trainee performance data.
 56. The method ofclaim 1, wherein: the trainer performance data comprising trainer wholebody position features, trainer body part position features, trainerdevice position features, or any combination thereof; the traineeperformance data comprise trainee whole body position features, traineebody part position features, trainee device position features, or anycombination thereof.
 57. The method of claim 1, further comprising:generating one or more avatars, providing the avatar with trainerperformance data and trainee performance data, and modifying the one ormore avatars based on the training program analysis and modifications.58. The method of claim 57, wherein the avatar is configured to changeform depending on the routine or task being performed and the traineeperformance of the routine or task.
 59. The method of claim 1, whereinthe training program is fully interactive using motion-based processing,hard select processing, gesture processing, voice command processing,neural command processing, or any combination thereof.
 60. The method ofclaim 1, further comprising: formatting all stored materials forefficient database storage, interaction, and retrieval.
 61. The methodof claim 1, further comprising: continuously, periodically, orintermittently analyzing all of the stored data; developing predictivetools or routines to assist trainees in performing the training program,routines, or tasks; developing trainee type predictive tools to assistspecific types of trainees in performing the training program, routines,or tasks; modifying one, some, or all aspects, features, attributes,properties, and/or characteristics of the training program, the trainingroutine, the training task, the environment or environments, hot spotsor interactive areas, or any combination thereof, during and/or afterthe trainer performing the generated training program or any partthereof; updating the training program, routines, tasks, and the hotspots; and storing the updates on the local databases, remote databases,or both.
 62. The method of claim 1, wherein, in the constructing orgenerating steps, the training program, routines, and/or tasks furthercomprising: explanatory sessions associated with the program, routinesand/or tasks; explanatory sessions associated with the equipment;question/answer sessions associated with the routines and/or tasks;question/answer sessions associated with the equipment; informationpresentation sessions associated with the program, routines and/ortasks; information presentation sessions associated with the equipment;trainee pass/fail explanatory sessions; trainee evaluation sessions;trainee performance ranking sessions; trainee feedback sessions; or anycombination thereof.
 63. The method of claim 1, further comprising:analyzing the trainer performance data, the trainee performance data,the differences, and/or the repeated trainee performance data, therepeated trainee differences, and the trainee completion data; andmodifying: (1) one, some, or all of the generated training programfeatures, attributes, properties, and characteristics; (2) one, some, orall of the generated training routine features, attributes, properties,and characteristics; (3) one, some, or all of the generated trainingtask features, attributes, properties, and characteristics; (4) one,some, or all of the generated program equipment features, attributes,properties, and characteristics; (5) one, some, or all of the hot spotsor interactive areas associated with the program content; (6) one, some,or all of the hot spots or interactive areas associated with the routinecontent; (7) one, some, or all of the hot spots or interactive areasassociated with the task content; (8) one, some, or all of the hot spotsor interactive areas associated with the equipment content, wherein themodifications improve the training program; and storing themodifications on the local databases, remote databases, or both.
 64. Themethod of claim 1, wherein, in the generating steps: the generated VRtraining program or the generated AR/MR/XR training program or thepreexisting VR training program or the preexisting AR/MR/XR trainingprogram comprising computer generated 2D, 3D, 4D, or nD components. 65.The method of claim 1, wherein the program content, routine content,task content, and equipment content independently comprising: textualcontent; non-textual content including: visual content; audio content;audiovisual content; haptic content; or any combination; or anycombination of textual content and non-textual content.
 66. The methodof claim 1, further comprising: analyzing trainer and trainee historicalperformance data; and modifying one, some, or all aspects, features,attributes, properties, and/or characteristics of the training program,the training routine, the training task, the environment orenvironments, hot spots or interactive areas, or any combinationthereof, during and/or after the trainer performing the generatedtraining program or any part thereof.